From f81d9d2b5cc5f5e69e208a1411d96553d761c591 Mon Sep 17 00:00:00 2001 From: Tudor Golubenco Date: Mon, 11 Nov 2024 18:49:11 +0200 Subject: [PATCH] Removing outdated blog posts --- announcing-go-sdk.mdx | 105 -- announcing-python-sdk-ga.mdx | 66 - auth-js.mdx | 367 ----- build-content-management-system.mdx | 451 ------ build-image-gallery-astro-cloudflare.mdx | 452 ------ building-protected-and-paywall.mdx | 969 ------------ cache-openai-langchain.mdx | 517 ------- chatgpt-on-your-data-2.mdx | 201 --- chatgpt-on-your-data.mdx | 101 -- data-modeling-in-typescript.mdx | 270 ---- datetime-picker.mdx | 300 ---- feature-flags-tutorial.mdx | 256 ---- file-attachments.mdx | 180 --- indexing-files-with-python.mdx | 452 ------ json-type.mdx | 319 ---- keyword-vs-semantic-search-chatgpt.mdx | 409 ----- launch-week-august-2023.mdx | 47 - navigating-many-to-one.mdx | 456 ------ new-getting-started-guides.mdx | 127 -- openapi-typesafe-react-query-hooks.mdx | 440 ------ rag-vector-sveltekit-xata.mdx | 420 ------ searching-for-retro-games.mdx | 494 ------ securely-querying-your-database.mdx | 101 -- semantic-search-openai-typescript-deno.mdx | 323 ---- server-actions-databases-data-handling.mdx | 180 --- travel-application-development-guide.mdx | 1592 -------------------- xata-trpc-clerk.mdx | 180 --- xata-workers-client-side-database.mdx | 143 -- xataform-is-here.mdx | 154 -- 29 files changed, 10072 deletions(-) delete mode 100644 announcing-go-sdk.mdx delete mode 100644 announcing-python-sdk-ga.mdx delete mode 100644 auth-js.mdx delete mode 100644 build-content-management-system.mdx delete mode 100644 build-image-gallery-astro-cloudflare.mdx delete mode 100644 building-protected-and-paywall.mdx delete mode 100644 cache-openai-langchain.mdx delete mode 100644 chatgpt-on-your-data-2.mdx delete mode 100644 chatgpt-on-your-data.mdx delete mode 100644 data-modeling-in-typescript.mdx delete mode 100644 datetime-picker.mdx delete mode 100644 feature-flags-tutorial.mdx delete mode 100644 file-attachments.mdx delete mode 100644 indexing-files-with-python.mdx delete mode 100644 json-type.mdx delete mode 100644 keyword-vs-semantic-search-chatgpt.mdx delete mode 100644 launch-week-august-2023.mdx delete mode 100644 navigating-many-to-one.mdx delete mode 100644 new-getting-started-guides.mdx delete mode 100644 openapi-typesafe-react-query-hooks.mdx delete mode 100644 rag-vector-sveltekit-xata.mdx delete mode 100644 searching-for-retro-games.mdx delete mode 100644 securely-querying-your-database.mdx delete mode 100644 semantic-search-openai-typescript-deno.mdx delete mode 100644 server-actions-databases-data-handling.mdx delete mode 100644 travel-application-development-guide.mdx delete mode 100644 xata-trpc-clerk.mdx delete mode 100644 xata-workers-client-side-database.mdx delete mode 100644 xataform-is-here.mdx diff --git a/announcing-go-sdk.mdx b/announcing-go-sdk.mdx deleted file mode 100644 index 3b613d76..00000000 --- a/announcing-go-sdk.mdx +++ /dev/null @@ -1,105 +0,0 @@ ---- -title: 'Announcing the release of the Xata Go SDK' -description: 'We are excited to announce the initial alpha version of the Xata Go SDK.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/announcing-xata-sdk-cover.png - alt: Xata Go SDK -author: Philip Krauss -date: 12-06-2023 -published: true -slug: announcing-xata-go-sdk -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/announcing-xata-sdk-cover.png ---- - -Earlier this year we shared a [community spotlight](https://xata.io/blog/community-spotlight-xata-go-sdk) focused on a Go SDK developed by a dedicated contributor, [xata-go](https://github.com/kerdokurs/xata-go). - -[Kerdo Kurs](https://www.linkedin.com/in/kerdokurs/), the developer behind this project, was a huge inspiration for us committing to an official Go SDK. Today, we're thrilled to announce the official release of our Xata Go SDK, and open the doors to the growing Go community šŸŽ‰ -The `v0.0.1` release features our most popular endpoints, enabling functionalities like searching a branch, using the ask endpoint for follow-up questions, and creating a database, among others. -You can see the evolving endpoint coverage in the ticket [xata-go#1](https://github.com/xataio/xata-go/issues/1). - -If you want to get started right away, head over to the Go SDK [documentation](/docs/sdk/go/overview) and browse the code on [GitHub](https://github.com/xataio/xata-go). - -In our documentation, we've included examples showing how to use the Go SDK, similar to the existing guides we have for TypeScript, Python, cURL, and SQL. - -## Anatomy of the SDK - -In the Go SDK, components are organized into specialized clients within distinct namespaces, like the `RecordsClient` for record-related functions. -All endpoints that are listed in our [API reference](/docs/api-reference/db/db_branch_name/transaction), are modeled within this client. - -The available clients are: - -- `BranchClient` -- `DatabasesClient` -- `FilesClient` -- `RecordsClient` -- `SearchAndFilterClient` -- `TableClient` -- `UsersClient` -- `WorkspaceClient` - -The code and corresponding test suites can be found in the [xata package](https://github.com/xataio/xata-go/tree/main/xata). - -If you're familiar with our SDKs, you'll notice familiar API naming patterns, adapted to align with Go's idiomatic style. -To illustrate the usage of the Go SDK, we curated a couple of examples. Browse [our docs](/docs/sdk/get) to see the full bandwidth of examples. - -The following example succinctly demonstrates, how to use the Ask endpoint. -For demo purposes we assume that the table `IMDB` contains all movies listed from [IMDB.com](https://www.imdb.com/) and we want to learn how many Ace Ventura movies exist. -More complex queries and follow up questions examples can be found [here](/docs/sdk/ask). - -```go -searchClient, _ := xata.NewSearchAndFilterClient() -result, _ := searchClient.Ask(context.TODO(), xata.AskRequest{ - TableName: "IMDB", - Question: "How many Ace Ventura movies are there?", -}) -``` - -In this example, we have not normalized data in multiple tables in one branch and want to search across this entire branch. -We are looking for the name _Philip_, but as there are multiple ways of writing Philip (such as _Filip_ or _Phillip_ or _Philippe_) we need to apply fuzziness to our search. `Fuzziness: xata.Int(2)` sets the search fuzziness level to a degree of 2, allowing for slight variations in the spelling of search terms. - -```go -searchClient, _ := xata.NewSearchAndFilterClient() -results, _ := searchClient.SearchBranch(context.TODO(), xata.SearchBranchRequest{ - Payload: xata.SearchBranchRequestPayload{ - Query: "Philip", - Fuzziness: xata.Int(2), - }, -}) -``` - -The last example illustrates the use of the [transaction](/docs/sdk/transaction) endpoint. -Transactions are a powerful way to do multiple operations in one go. -For the sake of the example, we solely want to create multiple records of actors that participated in the movie the Matrix. - -```go -recordsClient, _ := xata.NewRecordsClient() -result, _ := recordsClient.Transaction(context.TODO(), xata.TransactionRequest{ - Operations: []xata.TransactionOperation{ - xata.NewInsertTransaction(xata.TransactionInsertOp{ - Table: "Actors", - Record: map[string]any{ - "name": xata.String("Keanu Charles Reeves"), - "movie": xata.String("the_matrix"), - }, - Record: map[string]any{ - "name": xata.String("Carrie-Anne Moss"), - "movie": xata.String("the_matrix"), - }, - Record: map[string]any{ - "name": xata.String("Laurence Fishburne"), - "movie": xata.String("the_matrix"), - }, - }), - }, -}) -``` - -## What's next? - -The SDK is an alpha release, and not all [endpoints are covered](https://github.com/xataio/xata-go/issues/1), we continue to increase the API coverage of the SDK and provide bug fixes. - -We can't do it without you! If you want to contribute [code](https://github.com/xataio/xata-go), add an [example](https://github.com/xataio/xata-go/tree/main/examples) update the [docs](/docs/sdk/go/overview), or found a bug please open a PR or issue and we will assist you. All contributions are welcome āœ… - -To stay up to date with the latest at Xata, follow us on [X | Twitter](https://twitter.com/xata) or pop in and say hi in [Discord](https://xata.io/discord). - -Is there a language you wished we supported natively but don't today? Feel free to open up a [feature request](https://xata.canny.io/feature-requests) or check-in with our amazing community. diff --git a/announcing-python-sdk-ga.mdx b/announcing-python-sdk-ga.mdx deleted file mode 100644 index da744457..00000000 --- a/announcing-python-sdk-ga.mdx +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: 'Announcing General Availability of the Python SDK' -description: 'We are excited to ship the first generally available version of the Xata Python SDK.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/announcing-python-sdk-ga-cover.jpg - alt: Xata Python SDK -author: Philip Krauss -date: 08-29-2023 -published: true -slug: announcing-the-python-sdk-ga ---- - -At the start, Xata's initial focus was to make things better for developers using [Jamstack](https://xata.io/blog/jamstack-mern-lamp-stack-comparison#jamstack-and-databases). This led to us creating content that predominantly focused on providing a robust [TypeScript offering](https://www.npmjs.com/package/@xata.io/client). - -Motivated by our enthusiasm for connecting with developers, we soon launched a Python SDK, though it initially had just basic capabilities. Since that initial release, a few things have changed. The world has gone AI-crazy and Python has unofficially become the language of choice for AI/ML. This has coincided with our introduction of vector embeddings and the integration of OpenAI's ChatGPT for your data. - -As the Python SDK user base has growing steadily over the last few months, we have actively gathered feedback about what users like and dislike. - -Today, as part of our [Xata Launch week](https://xata.io/blog/launch-week-august-2023), we are thrilled to announce the [v1.0.0 release](https://pypi.org/project/xata/1.0.0/) of our [Python SDK](/docs/python-sdk/overview). - -## PEP-8 FTW! - -We received feedback that the SDK didn't feel pythonic, and we couldnā€™t agree more! Initially, in the `0.x` releases, the API was generated one-by-one from [our OpenAPI specification](/docs/rest-api#contexts), which resulted in a non-pythonic API. For this release, we have aligned as much as possible with the [PEP-8](https://peps.python.org/pep-0008/) standard. - -## Speed improvements - -Under the hood, we've made adjustments to how connections are managed and reused. The refactoring has yielded significant performance improvements across the board. The most notable ones include accelerated operation speeds: - -| Operation | Speedup (on average) | -| ------------------------------------ | -------------------- | -| Get a single record | 5.95x | -| Insert a single record | 4.95x | -| Insert 100 records with transactions | 2.22x | - -## How to migrate to 1.x? - -Migrating to a new major version can be tough due to the need to understand the changes that might break things and features that are no longer available. With that in mind, our goal was to minimize the impact while still implementing necessary changes for the greater good. You can check out the [full migration guide](/docs/python-sdk/migration-guide) in our docs. - -The most impactful user-facing change is the renaming of the API surface [[xata-py#93](https://github.com/xataio/xata-py/issues/93)] which has been acknowledged as a breaking change. Additionally, some API endpoint calls were simplified to remove unnecessary code bloat, like creating a database. - -Previously in `0.x`, you needed to investigate the payload shape and be specific about the region and branch name. - -```python -xata.databases().createDatabase("new_db", - { - "region": "us-east-1", - "branchName": "main", - } -) -``` - -In version `1.0.*`, we've redesigned the API interface to incorporate payload options directly into the method signature. We've also optimized the SDK's internal values to make assumptions to reuse the SDK internal values, like the `region`. This results in a more streamlined API structure, and you can achieve functionality using: - -```python -xata.databases().create("new_db") -``` - -## What happens to 0.x? - -We will continue providing support for the `0.x` version of the Xata Python SDK through maintenance releases for an additional year. During this time, new API enhancements and security fixes will be introduced; however, no backports of helpers or other improvements are planned. The `0.x` SDK version will be sunsetted by September 1st, 2024. - -## Whatā€™s next? - -Check out our [documentation](/docs/python-sdk/migration-guide) and let us know what you think. Weā€™d love to hear from you! If you think something is missing or you found a bug, open a ticket in the [xata-py](https://github.com/xataio/xata-py) repository. All contributions are welcome. You can also follow us on [Twitter](https://twitter.com/xata) or join us in [Discord](https://xata.io/discord). - -Is there a language you wished we supported natively but don't today? Feel free to open up a [feature request](https://xata.canny.io/feature-requests) or check-in with our amazing community. The [xata-go SDK](/blog/community-spotlight-xata-go-sdk) was initiated by [Kerdo](https://github.com/kerdokurs) and is hosted on [GitHub](https://github.com/kerdokurs/xata-go). [mrkresnofatih](https://github.com/mrkresnofatih) also added the [terraform-provider-xata](https://github.com/mrkresnofatih/terraform-provider-xata) to provision Xata. If you're interested in contributing, feel free to reach out with any questions! Happy building šŸ˜„ diff --git a/auth-js.mdx b/auth-js.mdx deleted file mode 100644 index 6977ba5d..00000000 --- a/auth-js.mdx +++ /dev/null @@ -1,367 +0,0 @@ ---- -title: 'Build Next.js apps with JWT authentication, Auth.js, and Xata' -description: 'Use Xata and Authā€¤js on two Nextā€¤js apps: app directory and pages directory.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/xata-authjs-nextjs.png - alt: Xata JWT and Auth.js logos -author: Attila Fassina -date: 03-23-2023 -tags: ['auth'] -published: true -slug: xata-authjs-nextjs ---- - -In this article we will explore how to get authentication working in your Next.js app using Auth.js and Xata. It's important to note that Next.js has at the moment two routing architectures that are defined via your project's file system: - -- App: where routes are mapped in `/app` directory -- Pages: where routes are mapped in `/pages` directory - -Working the conditions and conventions of each directory can get you with any routes landscape. Though there are fundamental differences on how you will think the data story depending on which architecture you choose. If you've been working (or has worked) with Next.js at any point until now, chances are you're familiar with the pages directory architecture. And in case you haven't exactly kept track to the new experimental releases since Next.js 13, there will be some catch-up to do before you start using and [understanding the App Directory](https://www.smashingmagazine.com/2023/02/understanding-app-directory-architecture-next-js/). - -Before going into the Next.js specifics, let's first understand our goals with this project. Visualizing the mental model is an important step to ensure your user's data is switching hands as little as possible. - -## Defining a mental model - -A diagram is probably the fastest way to visualizing a process. This is important to see if there aren't any redundant calls, or if any of those requests are chained in some condition. Chained requests are the grim reapers of perceived performance, we need to get those out of the way as soon as possible. - - - -In the above graph, we have ā€œFrontendā€ and ā€œBackendā€ that are both handled by Next.js. We have the ā€œauth providerā€ which our app is connected to via Auth.js (within a Next.js endpoint), and we have Xata as our database, connecting as well via Next.jsā€™ backend layer. We can examine and translate the flow of that diagram into 3 steps: - -1. **Authentication** - Provider performs user authentication, the library creates a `session` and the app receives that as a `payload` from the auth request. -2. **Resource Request** - Authenticated User performs request with `resourceId`, app takes `userId` from `session` -3. **Granting Access** - Xata filters all resources from table to only the ones owned by `userId` and returns (if exists) the one with `resourceId` - -Now that we aligned ourselves about the target implementation, it's time to roll up our sleeves and start writing some code. First, let's prepare the authentication endpoint. - -## Setup Auth - -When working with Auth.js and Next.js, everything starts with a very special API Route. We create a dynamic catch-all route: `api/[...nextauth].ts`. There's a lot to unpack on this naming pattern for first timers, let's have a quick look: - -- Everything inside `/pages/api/auth` is a resource route in Next.js, it will either be a Serverless Function or an Edge Function depending on the runtime you choose. -- Filenames surrounded by square brackets (`[]`) indicate this route is dynamic, so it will take the corresponding segment path and add to the search params of the route. E.g.: `[slug].js` will serve `[serverless.com/xata]()` and receive `{ slug: "xata" }`as a parameter. -- By adding the spread (`ā€¦`) to the param, we indicate we take all segment paths. So not only `/api/xata` but also `/api/xata/auth/another/route` will be rendered by that route. Hence why we namespace our catch-all route inside `/auth` - -> There is currently an [opened issue within the Auth.js repository](https://github.com/nextauthjs/next-auth/issues/6792) to support the new [Route Handlers](https://beta.nextjs.org/docs/routing/route-handlers), if your intention is going full `/app` directory, keep an eye on it. Otherwise, everything will work just fine keeping the API Route inside `/pages`. - -Our Serverless Route will then import `NextAuth` as a high order function, this will handle the necessary routes the library will need to implement and we will only pass the configuration object to it. Xata has an adapter, which means once the authentication is done, Auth.js will itself create an entry into a Xata database for that user, so the authentication provider will only be used for registering the user and sign-in. Auth.js has a big number of authentication providers, for this example we will use GitHub. Your `/pages/api/auth/[...nextauth].ts` will then look like this: - -```tsx -import NextAuth from 'next-auth'; -import GitHubProvider from 'next-auth/providers/github'; -import { XataAdapter } from '@next-auth/xata-adapter'; -import { XataClient } from '~/shared/xata.codegen'; - -const client = new XataClient(); - -export const authConfig = { - adapter: XataAdapter(client), - providers: [ - GitHubProvider({ - clientId: process.env.GITHUB_ID, - clientSecret: process.env.GITHUB_SECRET - }) - ] -}; - -export default NextAuth(authConfig); -``` - -With that, Auth.js is ready to authenticate and provide our app with a valid session to fetch user specific data. Now it's time to setup our database our app can communicate to Xata, it is important to setup a workspace and push our schema to it. First ,take the following `json` and save it somewhere: - -```json -{ - "tables": [ - { - "name": "nextauth_users", - "columns": [ - { - "name": "email", - "type": "email" - }, - { - "name": "emailVerified", - "type": "datetime" - }, - { - "name": "name", - "type": "string" - }, - { - "name": "image", - "type": "string" - } - ] - }, - { - "name": "nextauth_accounts", - "columns": [ - { - "name": "user", - "type": "link", - "link": { - "table": "nextauth_users" - } - }, - { - "name": "type", - "type": "string" - }, - { - "name": "provider", - "type": "string" - }, - { - "name": "providerAccountId", - "type": "string" - }, - { - "name": "refresh_token", - "type": "string" - }, - { - "name": "access_token", - "type": "string" - }, - { - "name": "expires_at", - "type": "int" - }, - { - "name": "token_type", - "type": "string" - }, - { - "name": "scope", - "type": "string" - }, - { - "name": "id_token", - "type": "text" - }, - { - "name": "session_state", - "type": "string" - } - ] - }, - { - "name": "nextauth_verificationTokens", - "columns": [ - { - "name": "identifier", - "type": "string" - }, - { - "name": "token", - "type": "string" - }, - { - "name": "expires", - "type": "datetime" - } - ] - }, - { - "name": "nextauth_users_accounts", - "columns": [ - { - "name": "user", - "type": "link", - "link": { - "table": "nextauth_users" - } - }, - { - "name": "account", - "type": "link", - "link": { - "table": "nextauth_accounts" - } - } - ] - }, - { - "name": "nextauth_users_sessions", - "columns": [ - { - "name": "user", - "type": "link", - "link": { - "table": "nextauth_users" - } - }, - { - "name": "session", - "type": "link", - "link": { - "table": "nextauth_sessions" - } - } - ] - }, - { - "name": "nextauth_sessions", - "columns": [ - { - "name": "sessionToken", - "type": "string" - }, - { - "name": "expires", - "type": "datetime" - }, - { - "name": "user", - "type": "link", - "link": { - "table": "nextauth_users" - } - } - ] - } - ] -} -``` - -And now if you have our CLI installed, you can use the `xata` namespace, otherwise `npx @xata.io/cli@latest` will also work: - -```bash -xata init --schema="path/to/schema.json" - -``` - -Finally, all code is in place to: - -1. Connect to GitHub OAuth -2. Create a session within our Next.js app -3. Push user registration to our Xata database - -What we need now is a set of keys to effectively connect to those services in a secure way. - -## Configure environment variables - -The environment variables `GITHUB_ID` and `GITHUB_SECRET` are provided by GitHub once you register an [OAuth App](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app). For your code to run properly you will need to have a `.env` (or `.env.local`) file looking like: - -```bash -# Xata specifics -XATA_API_KEY= -XATA_BRANCH=main - -# GitHub OAuth specifics -GITHUB_ID= -GITHUB_SECRET= - -# Auth.js specifics -NEXTAUTH_URL=http://localhost:3000 # if in local dev -NEXTAUTH_SECRET= -``` - -Finally, it's time to manage some data. - -## Fetching user data - -There are a few ways with which you can provide secure access to routes in your Auth.js app. One is via the Auth.js middleware, this will redirect anyone without a `session` automatically to the login route you set up. This approach should work in both App Directory and Pages Directory. - -If working with the Pages Directory, you must set the `` encapsulating routes that should have access to the Session (if using `getStaticProps`), or check the session within `getServerSideProps` and pass it down as render props. - -Lastly, in the app directory, the best way is to use `getServerSession` method, which will retrieve the method in every Server Component that needs to use it. This is close to what `getServerSideProps` can do, though it's on a component level - while on `getServerSideProps`we're dealing at route level. - -When working with React Server components, don't worry about firing multiple requests, it always hit the same endpoint and therefore Next.js will be able to deduplicate it, so there will only be one fetch request. - -## Accessing user specific data - -Once we got Auth setup, we can treat Row-Level Security (RLS) as an if/else statement. For example, take this contrived profile page: - -```tsx -import { fetchUserData } from '~/lib/db.server.ts' -import { getSession } from 'next-auth/react' - -export default function Profile({ user, userData }) { - return ( /** React component **/ ) -} - -export const getServerSideProps = async ({ - req, - res, -}) => { - const session = await getSession({ req }) - - return { - props: { - user: session?.user, - todos: await fetchUserData(session?.user.email), - }, - } -} -``` - -In the above example, we are using the user's email as their unique identifier (or primary key). And we implemented a getter in a file named `db.server.ts`. Let's see how that looks like using the Xata SDK. - -```tsx -// the file generated via our CLI -import { getXataClient } from '~/lib/xata.codegen.server'; - -const xata = getXataClient(); - -export const fetchUserData = async (email: string) => { - const userData = await xata.db.itemList - .filter({ - user: { - email - } - }) - .sort('created_at', 'desc') - .getPaginated({ pagination: { size: 100, offset: 0 } }); - - return userData; -}; -``` - -Remember, this method may **only** be accessed from the server-side. In Next.js this means: - -- from a React Server Component -- from `getStaticProps`, `getStaticPaths`, or `getServerSideProps` -- from a Middleware, Serverless Function, or Edge Function - -To retrieve data from the client-side, we would need a different method that sends a `POST` request to an API Route or a Route Handler, for example: - -```tsx -export const fetchData = async (userEmail: string) => { - const response = await fetch('/api/fetch-data', { - method: 'post', - body: JSON.stringify({ - userEmail - }) - }); - - if (response.ok) { - return response; - } else { - const err = await response.json(); - throw new Error(`ERROR [${response.status}]: ${err.message ?? 'The server returned an error'}`); - } -}; -``` - -## Working code - -Now that we are done, you can compare your code or check the components that we wrote to use this logic in our examples repository: - -- [App Directory example](https://github.com/xataio/examples/tree/main/apps/sample-next-auth-app). -- [Pages Directory example](https://github.com/xataio/examples/tree/main/apps/sample-next-auth-pages). - -## Conclusion - -I hope this article has helped you getting started with Row-Level Security and got you through the door with using Auth in a Next.js app with no vendor lock-in and owning your data. If any questions remain, reach Atila both on our [discord](https://xata.io/discord) or on [twitter](https://twitter.com/intent/follow?screen_name=atilafassina). diff --git a/build-content-management-system.mdx b/build-content-management-system.mdx deleted file mode 100644 index be5f9eff..00000000 --- a/build-content-management-system.mdx +++ /dev/null @@ -1,451 +0,0 @@ ---- -title: 'Create your own content management system with Remix and Xata' -description: 'Learn how to create a custom CMS using Xata, Remix, Novel, LiteLLM, and Vercel.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/cms-cover.jpeg - alt: Xata CMS -author: Rishi Raj Jain -date: 02-07-2024 -published: true -slug: content-management-system-remix-xata -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/cms-cover.jpeg ---- - -In this post, you'll create a content CMS using Xata, Remix, Novel, LiteLLM, and Vercel. You'll learn how to: - -- Set up Xata -- Create a schema with different column types -- Handle forms in Remix using Form Actions -- Implement Client Side Image Uploads -- Use an AI-powered WYSIWYG Editor -- Implement content-wide search -- Create dynamic content routes with Remix - -## Before you begin - -### Prerequisites - -You'll need the following: - -- A [Xata](https://xata.io/) account -- [Node.js 18](https://nodejs.org/en/blog/announcements/v18-release-announce) or later -- An [OpenAI](https://platform.openai.com) account -- A [Vercel](https://vercel.com) Account - -### Tech Stack - -| Technology | Description | -| --------------------------------------------- | ------------------------------------------------------------------------------- | -| [Xata](https://xata.io) | Serverless database platform for scalable, real-time applications. | -| [Remix](https://remix.run) | Framework for building full-stack web applications with focus on Web Standards. | -| [LiteLLM](https://github.com/BerriAI/litellm) | Call all LLM APIs using the OpenAI format. | -| [Novel](https://novel.sh) | A Notion-style WYSIWYG editor with AI-powered autocompletion | -| [TailwindCSS](https://tailwindcss.com/) | CSS framework for building custom designs | -| [Vercel](https://vercel.com) | A cloud platform for deploying and scaling web applications. | - -## Setting up a Xata Database - -After you've created a Xata account and are logged in, create a database. - -![Create a database](/images/cms-remix-xata-01.png) - -The next step is to create a table, in this instance `uploads`, that contains all the uploaded images. - -![Create uploads table](/images/cms-remix-xata-02.png) - -Great, now click on **Schema** in the left sidebar and create one more table `content`. You can do this by clicking **Add a table**. The tables will contain user content and user uploaded photographs. With that completed, you will see the schema as below. - -![View uploads and content schema](/images/cms-remix-xata-04.png) - -Letā€™s move on to adding relevant columns in the tables you've just created. - -## Creating the Schema - -In the `uploads` table, you want to store all the images only (and no other attributes) so that you can create references to the same image object again, if needed. - -Proceed with adding the column named `image`. This column is responsible for storing the `file` type objects. In our case, the `file` type object is for images, but you can use this for storing any kind of blob (e.g. PDF, fonts, etc.) thatā€™s sized up to 1 GB. - -First, click **+ Add column** and select **File**. - -![Add a column](/images/cms-remix-xata-05.png) - -Set the column name to `photo` and to make files public (so that they can be shown to users when they visit the image gallery), check the **Make files public by default** option. - -![Make files public by default](/images/cms-remix-xata-06.png) - -In the `content` table, we want to store the attributes such as contentā€™s unique slug (the path of the url where content will be displayed), title, author name, authorā€™s image with itā€™s dimensions, and contentā€™s og image with itā€™s dimensions. - -Proceed with adding the column named `slug`. It is responsible for maintaining the uniqueness of each content that gets created. Click **+ Add a column**, select `String` type and enter the column name as `slug`. To associate a slug with only one content, check the `Unique` attribute to make sure that duplicate entries do not get inserted. - -![Add slug column](/images/cms-remix-xata-07.png) - -In similar fashion, create `title`, `author_name`, `author_image_url`, `og_image_url`, `author_image_w`, `author_image_h`, `og_image_w`, `og_image_h` as `String` type (but not `Unique`). - -Great, you can also store the user `content` as `Text` type. While `String` is a great default type, storing more than 2048 characters would require you to switch to the `Text` type. Read more about the limits in [Xata Column limits](https://xata.io/docs/rest-api/limits#column-limits). - -Lovely! With all that done, the final schema shall be as below šŸ‘‡šŸ» - -![Add more CMS columns](/images/cms-remix-xata-08.png) - -## Setting up the project - -Clone the app repository and follow this tutorial; you can fork the project by running the following command: - -```bash -git clone https://github.com/rishi-raj-jain/remix-wysiwyg-litellm-xata-vercel -cd remix-wysiwyg-litellm-xata-vercel -npm install -``` - -## Configure Xata with Remix - -To seamlessly use Xata with Remix, install the Xata CLI globally: - -```bash -npm install @xata.io/cli -g -``` - -Then, authorize the Xata CLI so it is associated with the logged in account: - -```bash -xata auth login -``` - -![Create new API key](/images/cms-remix-xata-09.png) - -Great! Now, initialize your project locally with the Xata CLI command. In this command, you will need to use the database URL for the database that you just created. You can copy the URL from the Settings page of the database. - -```bash -xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.us-east-1.xata.sh/db/remix-wysiwyg-litellm-xata-vercel -``` - -Answer some quick one-time questions from the CLI to integrate with Remix. - -![Xata CLI](/images/cms-remix-xata-10.png) - -## Implementing form actions in Remix - -With Remix, [Route Actions](https://reactrouter.com/en/main/route/action) are the way to process form `POST` request(s). Hereā€™s how weā€™ve enabled form actions to process the form submissions and insert records into the Xata database. - -```tsx -import { Form } from '@remix-run/react' -import { ActionFunctionArgs, json, redirect } from '@remix-run/node' - -export async function action({ request }: ActionFunctionArgs) { - // Get the form data - const body = await request.formData() -} - -export default function Index() { - return ( -
- {% my form elements %} -
- ) -} -``` - -This allows you to colocate the serverless backend and frontend flow for a given page in Remix. Say, you accept a form submission containing the title, slug, and the contentā€™s HTML, process it on the server, and sync it with your Xata serverless database. Hereā€™s how youā€™d do all of that in a single Remix route (`app/routes/_index.tsx`). - -```tsx -// app/routes/_index.tsx - -import { Editor } from 'novel'; -import { Form } from '@remix-run/react'; -import { getXataClient } from '@/xata.server'; -import Upload from '@/components/Utility/Upload'; -import { ActionFunctionArgs, json, redirect } from '@remix-run/node'; - -export async function action({ request }: ActionFunctionArgs) { - // Import the Xata Client created by the Xata CLI in app/xata.server.ts - const xata = getXataClient(); - // Get the form data - const body = await request.formData(); - const slug = body.get('slug') as string; - const title = body.get('title') as string; - const content = body.get('content-html') as string; - // Sync the attributes to the content table in Xata - await xata.db.content.create({ slug, title, content }); -} - -export default function Index() { - return ( -
- New Article - Title - - Content - - Slug - - -
- ); -} -``` - -## Handling Client Side Image Uploads with Xata - -To let user add their own custom OG Image with the content, we use Xata [Upload URLs](https://xata.io/docs/sdk/file-attachments#upload-urls) to handle image uploads on the client side. There are 2 steps to make a successful client side image upload with Xata and Remix: - -1. Create a record with empty photo `base64Content` and obtain the photoā€™s **uploadUrl**. - -```tsx -// app/routes/api_.image.upload.tsx - -import { json } from '@remix-run/node'; -import { getXataClient } from '@/xata.server'; - -export async function loader() { - const xata = getXataClient(); - // Use the Xata client to create a new 'photo' record with an empty base64 content - const result = await xata.db.uploads.create({ photo: { base64Content: '' } }, ['photo.uploadUrl']); - return json({ uploadUrl: result?.photo?.uploadUrl }); -} -``` - -2. Do a client side `PUT` request to the **uploadUrl** with body as imageā€™s buffer. - -```tsx -// app/components/Utility/Upload.tsx - -const uploadFile = (e: ChangeEvent) => { - // Get the reference to the file uploaded - const file = e.target.files?.[0]; - if (!file) return; - const reader = new FileReader(); - reader.onload = async (event) => { - // Load the file buffer - const fileData = event.target?.result; - if (fileData) { - // Create blob from the file data with the relevant file's type - const body = new Blob([fileData], { type: file.type }); - // Make a fetch to the get the uploadUrl - fetch('/api/image/upload') - .then((res) => res.json()) - .then((res) => { - // Use the uploadUrl to upload the buffer - fetch(res.uploadUrl, { - body, - method: 'PUT' - }); - }); - } - }; - // Read the user uploaded file as buffer - reader.readAsArrayBuffer(file); -}; -``` - -## Using an AI powered WYSIWYG Editor - -For making it easier to write content, users need a reliable and user-friendly AI powered WYSIWYG editor. Weā€™re using Novel, a Notion-Style WYSIWYG Editor providing a seamless experience with intuitive features and real-time preview of the content being written. To get the content being written as HTML, we use Novelā€™s `onUpdate` callback and set the HTML string to an input inside the form element. - -```tsx -// app/routes/_index.tsx - -import { Editor } from 'novel'; -import { Form } from '@remix-run/react'; - -export default function Index() { - return ( -
- Content - - { - if (!e) return; - const tmp = e.getHTML(); - const htmlSelector = document.getElementById('content-html'); - if (tmp && htmlSelector) htmlSelector.setAttribute('value', tmp); - }} - /> - - - ); -} -``` - -## Implementing autocompletion using LiteLLM - -Under the hood, Novel makes a POST request to `/api/generate` expecting a stream of tokens from OpenAI API. Well, letā€™s see how weā€™ve customised the endpoint to get the flexibility of using any AI API provider with LiteLLM. With LiteLLM, you can call 100+ LLMs with the same OpenAI-like input and output. To implement autocompletion with streaming, we use the `completion` method with `stream` flag set to `true` and further return the response obtained as a [ReadableStream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream). - -```tsx -// app/routes/api_.generate.tsx - -import { completion } from 'litellm'; -import { ActionFunctionArgs } from '@remix-run/node'; - -export async function action({ request }: ActionFunctionArgs) { - const encoder = new TextEncoder(); - const { prompt } = await request.json(); - const response = await completion({ - n: 1, - top_p: 1, - stream: true, - temperature: 0.7, - presence_penalty: 0, - model: 'gpt-3.5-turbo', - messages: [ - { - role: 'system', - content: - 'You are an AI writing assistant that continues existing text based on context from prior text. ' + - 'Give more weight/priority to the later characters than the beginning ones. ' + - 'Limit your response to no more than 200 characters, but make sure to construct complete sentences.' - // we're disabling markdown for now until we can figure out a way to stream markdown text with proper formatting: https://github.com/steven-tey/novel/discussions/7 - // "Use Markdown formatting when appropriate.", - }, - { - role: 'user', - content: prompt - } - ] - }); - // Create a streaming response - const customReadable = new ReadableStream({ - async start(controller) { - for await (const part of response) { - try { - const tmp = part.choices[0]?.delta?.content; - if (tmp) controller.enqueue(encoder.encode(tmp)); - } catch (e) { - console.log(e); - } - } - controller.close(); - } - }); - // Return the stream response and keep the connection alive - return new Response(customReadable, { - // Set the headers for Server-Sent Events (SSE) - headers: { - Connection: 'keep-alive', - 'Content-Encoding': 'none', - 'Cache-Control': 'no-cache, no-transform', - 'Content-Type': 'text/event-stream; charset=utf-8' - } - }); -} -``` - -## **Implementing Content Wide Search with Xata Search** - -To let user search through the entire collection of the content, we use Remix Route Actions with Xata Search to retrieve relevant records from the database. With Xata Search, you can choose the tables to **search through**, in this instance, `content` and set the targets to **search on**, in this instance, `title`, `slug`, `content` and `author_name`. - -```tsx -// app/routes/_index.tsx - -export async function action({ request }: ActionFunctionArgs) { - const body = await request.formData(); - const search = body.get('search') as string; - // If the 'search' parameter is missing, redirect to '/content' - if (!search) return redirect('/content'); - const xata = getXataClient(); - - // Use the Xata client to perform a search across specified tables with fuzziness - const { records } = await xata.search.all(search, { - tables: [ - { - table: 'content', - target: ['content', 'title', 'slug', 'author_name'] - } - ], - fuzziness: 2 - }); - - // Extract the 'record' property from each search result containing the content - const result = records.map((i) => i.record); - return json({ search, result }); -} -``` - -## Creating Dynamic Routes in Remix - -To create a page dynamically for each content, we're gonna use Remix Dynamic Routes and Route Loaders. Creating a page with **$** in it, in this instance, content\_.**$id**.tsx specifies a dynamic route where each part of the URL for e.g. for `/content/a`, `/content/b` or `/content/anything` captures the last segment into the **id** param. - -With Remix Loader and Xata Records, we dynamically query the database to give us the content pertaining to a particular id. Once obtained, we process and return the content as HTML string. Finally, we use the loader data to prototype the UI with best practices such as lazy loading non-critical images. - -```tsx -// app/routes/content_.$id.tsx - -import { getXataClient } from '@/xata.server'; -import Image from '@/components/Utility/Image'; -import { useLoaderData } from '@remix-run/react'; -import { unescapeHTML } from '@/lib/util.server'; -import { getTransformedImage } from '@/lib/ast.server'; -import { LoaderFunctionArgs, redirect } from '@remix-run/node'; - -export async function loader({ params }: LoaderFunctionArgs) { - if (!params.id) return redirect('/404'); - const xata = getXataClient(); - // Use the Xata client to fetch content from the 'content' table based on the 'slug' - const content = await xata.db.content - .filter({ - slug: params.id - }) - .getFirst(); - if (content) { - const output = await getTransformedImage(content); - return { ...content, content: unescapeHTML(output) }; - } - // If content is not found, redirect to '/404' - return redirect('/404'); -} - -export default function Pic() { - const content = useLoaderData(); - return ( -
- {content.title} -
- {content.author_name} -
- {content.author_name} -
-
- {content.title} - {content.content &&
} -
- ); -} -``` - -## Deploy to Vercel - -The repository, is now ready to deploy to Vercel. Use the following steps to deploy: šŸ‘‡šŸ» - -- Start by creating a GitHub repository containing your app's code. -- Then, navigate to the Vercel Dashboard and create a **New Project**. -- Link the new project to the GitHub repository you just created. -- In **Settings**, update the _Environment Variables_ to match those in your local `.env` file. -- Deploy! šŸš€ - -## More Information - -For more detailed insights, explore the references cited in this post. - -| Resource | Link | -| --------------------- | ----------------------------------------------------------------------- | -| GitHub Repo | https://github.com/rishi-raj-jain/remix-wysiwyg-litellm-xata-vercel | -| Remix with Xata | https://xata.io/docs/getting-started/remix | -| Xata File Attachments | https://xata.io/docs/sdk/file-attachments#upload-a-file-using-file-apis | -| Remix Route Actions | https://remix.run/docs/en/main/discussion/data-flow#route-action | -| Remix Route Loaders | https://remix.run/docs/en/main/discussion/data-flow#route-loader | - -## Whatā€™s next? - -We'd love to hear from you if you have any feedback on this tutorial, would like to know more about Xata, or if you'd like to contribute a community blog or tutorial. Reach out to us on [Discord](https://discord.com/invite/kvAcQKh7vm) or join us on [X | Twitter](https://twitter.com/xata). Happy building šŸ¦‹ diff --git a/build-image-gallery-astro-cloudflare.mdx b/build-image-gallery-astro-cloudflare.mdx deleted file mode 100644 index fc64d0a6..00000000 --- a/build-image-gallery-astro-cloudflare.mdx +++ /dev/null @@ -1,452 +0,0 @@ ---- -title: 'Build your own image gallery CMS' -description: 'Learn how to create a custom image gallery CMS using Xata, Astro, and Cloudflare Pages.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/gallery-cms-astro-cloudflare.png - alt: Xata image gallery -author: Rishi Raj Jain -date: 01-04-2024 -tags: ['fpFileAttachments'] -published: true -slug: build-image-gallery-astro-cloudflare -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/gallery-cms-astro-cloudflare.png ---- - -In this post, you'll create an image gallery CMS using Astro, Xata, and Cloudflare Pages. You'll learn how to: - -- Set up Xata -- Create a schema with different column types -- Resize and blur images -- Fetch all records without pagination -- Handle forms in Astro using view transitions - -## Before you begin - -### Prerequisites - -You'll need the following: - -- A [Xata](https://app.xata.io/signin) account -- [Node.js 18](https://nodejs.org/en/download) or later -- A [Cloudflare](https://workers.cloudflare.com/) account - -### Tech stack - -| Technology | Description | -| --------------------------------------------------- | ----------------------------------------------------------------------------- | -| [Xata](https://xata.io) | Serverless database platform for scalable, real-time applications. | -| [Astro](https://astro.build) | Framework for building fast, modern websites with serverless backend support. | -| [Tailwind CSS](https://tailwindcss.com/) | CSS framework for building custom designs | -| [Cloudflare Pages](https://workers.cloudflare.com/) | Platform for deploying and hosting web apps with global distribution. | - -## Setting up a Xata database - -After you've created a Xata account and are logged in, create a database. - -![Create a database](/images/build-image-gallery/image1.png) - -The next step is to create a table, in this instance `uploads`, that contains all the uploaded images. - -![Create a table](/images/build-image-gallery/image2.png) - -Great! Now, click **Schema** in the left sidebar and create two more tables `profiles` and `photographs`. You can do this by clicking **Add a table**. The newly created tables will contain user profile data and user uploaded photograph(s) data respectively. - -![Newly created table](/images/build-image-gallery/image3.png) - -With that completed, you will see the schema. - -![Schema displayed](/images/build-image-gallery/image4.png) - -Letā€™s move on to adding relevant columns in the tables you've just created. - -## Creating the schema - -In the `uploads` table, you want to store all the images only (and no other attributes) so that you can create references to the same image object again, if needed. - -Proceed with adding the column named `image`. This column is responsible for storing the `file` type objects. In our case, the `file` type object is for images, but you can use this for storing any kind of blob (e.g. PDF, fonts, etc.) thatā€™s sized up to 1 GB. - -First, click **+ Add column** and select **File**. - -![Add column](/images/build-image-gallery/image5.png) - -Set the column name to `image` and to make files public (so that they can be shown to users when they visit the image gallery), check the **Make files public by default** option. - -![Make files public](/images/build-image-gallery/image6.png) - -In the `profiles` table, we want to store the attributes such as userā€™s unique slug (the path of the url where gallery will be displayed of that user), their name, their image with itā€™s dimension, and itā€™s transformed imageā€™s base64 hash. Youā€™ll reap the benefits of storing the hash to create literally 0 Cumulative Layout Shift (CLS) page(s). - -Proceed with adding the column named `slug`. It is responsible for maintaining the uniqueness of each profile that gets created. Click **+ Add a column**, select `String` type and enter the column name as `slug`. To associate a slug with only one user, check the `Unique` attribute to make sure that duplicate entries do not get inserted. - -![Add slug column](/images/build-image-gallery/image7.png) - -In similar fashion, create `name`, `image`, `height` and `width` columns as `String` type (but not `Unique`). - -Great, you can also store `imageHash` as `Text` type so that you can instantly retrieve the imageā€™s blur hash sized up to `200 KB`. While `String` is a great default type, storing more than 2048 characters would require you to switch to the `Text` type. Read more about the limits in [Xata Column limits](https://xata.io/docs/rest-api/limits#column-limits). - -Click **+ Add a column** and select the `Text` type. - -![Add a column](/images/build-image-gallery/image8.png) - -Enter the column name as `imageHash` and press `Create column`. - -![Enter the column name](/images/build-image-gallery/image9.png) - -Much similar what we did above, in the `photographs` table, we create `name`, `tagline`, `image`, `height`, `width`, `profile-slug`, and `slug` as `String` type and `imageHash` as the `Text` type column. The columns `slug` and `profile-slug` refer to photographā€™s and user profileā€™s slug respectively. - -Lovely! With all that done, the final schema will look something like the following... - -![Final schema](/images/build-image-gallery/image10.png) - -## Setting up your project - -Clone the app repository and follow this tutorial; you can fork the project by running the following command: - -```bash -git clone https://github.com/rishi-raj-jain/image-gallery-cms-with-astro-xata-cloudflare -cd image-gallery-cms-with-astro-xata-cloudflare -pnpm install -``` - -## Configure Xata with Astro - -To seamlessly use Xata with Astro, install the Xata CLI globally: - -```bash -npm install @xata.io/cli -g -``` - -Then, authorize the Xata CLI so it is associated with the logged in account: - -```bash -xata auth login -``` - -![Authorize the CLI](/images/build-image-gallery/image11.png) - -Great! Now, initialize your project locally with the Xata CLI command: - -```bash -xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.ap-southeast-2.xata.sh/db/image-gallery-cms-with-xata-astro-cloudflare -``` - -Answer some quick one-time questions from the CLI to integrate with Astro. - -![Xata init with Astro project](/images/build-image-gallery/image12.png) - -## Implementing form actions in Astro - -You can also allow transitions on form submissions by adding the `ViewTransitions` component. -Hereā€™s an example of the enabled form actions with view transitions in `src/layouts/Layout.astro`: - -```astro ---- -// File: src/layouts/Layout.astro - -import { ViewTransitions } from "astro:transitions"; ---- - - - - - - - - - - -``` - -This allows you to colocate the backend and frontend flow for a given page in Astro. Say, you accept a form submission containing the name, slug, and the image URL of the user, process it on the server to generate a blur Base64 hash, and sync it with your Xata serverless database. Hereā€™s how youā€™d do all of that in a single Astro route (`src/pages/profile/create.astro`). - -```astro ---- -// File: src/pages/profile/create.astro - -const response = { form: false, message: '', created: false, redirect: null } - -// ... - -if (Astro.request.method === 'POST') { - try { - // Indicate that the request is being processed - response.form = true - - // Get the user email from the form submissions - const data = await Astro.request.formData() - - // Get the user slug, name, and image: URL, width, and height from the form submissions - const userSlug = data.get('slug') as string - const userName = data.get('name') as string - const userImage = data.get('custom_upload_user__uploaded_image_url') as string - const userImageW = data.get('custom_upload_user__uploaded_w') as string - const userImageH = data.get('custom_upload_user__uploaded_h') as string - - // Create a blur url of the user image - - // Create the user record with the slug - - // Redirect user to the next step - } catch (e) { - // pass - } -} ---- - -
- - - - - -``` - -## Handling image uploads server-side with the Xata SDK - -![`/photograph/create` - where users upload their photographs](/images/build-image-gallery/image13.png) - -As Cloudflare Pages request body size allows up to 100 MB, youā€™ll be able to handle image uploads on the server-side. Create an Astro endpoint (`src/pages/api/upload/index.ts`) to receive POST requests containing image binaries and use the Xata SDK to store them in the `uploads` table. - -After doing sanity checks on the request body, first create a new (empty) record in your `uploads` table, and then use it as a reference to place the image (buffer) using the Xata TypeScript SDK. Once successfully completed, the endpoint responds with the imageā€™s `public URL`, `height` and `width` back to the front-end to include in the form fields. - -```ts -// File: src/pages/api/upload/index.ts - -import { json } from '@/lib/response'; -import { getXataClient } from '@/xata'; -import type { APIContext } from 'astro'; - -// Import the Xata Client created by the Xata CLI in src/xata.ts -const xata = getXataClient(); - -export async function POST({ request }: APIContext) { - const data = await request.formData(); - const file = data.get('file'); - - // Do sanity checks on file - - try { - // Obtain the uploaded file as an ArrayBuffer - const fileBuffer = await file.arrayBuffer(); - - // Create an empty record in the uploads table - const record = await xata.db.uploads.create({}); - // Using the id of the record, insert the file using upload method - await xata.files.upload({ table: 'uploads', record: record.id, column: 'image' }, fileBuffer, { - mediaType: file.type - }); - - // Read the inserted image - const { image } = await xata.db.uploads.read(record.id); - - // Destructure its dimension and public URL - const { url, attributes } = image; - const { height, width } = attributes; - return json({ height, width, url }, 200); - } catch (error) { - // Handle errors - } -} -``` - -## Using Xata image transformations to create blurred images - -Once a user submits their profile on the `/profile/create` page, before creating a record in the `profiles` table, generate a Base64 buffer of their blurred images. To create blurred images from the originals, use Xata image transformations. With Xata image transformations, youā€™re able to request an on-demand public URL which resizes the image to given height and width, and blur the image. In this particular example, you can resize the image to 100 x 100 dimensions and blur it up to 75% from the original. - -```astro ---- -// File: src/pages/profile/create.astro - -// Import the Xata Client created by the Xata CLI in src/xata.ts -import { getXataClient } from '@/xata' - -// Import the transformImage function by Xata Client -import { transformImage } from '@xata.io/client' - -const response = { form: false, message: '', created: false, redirect: null } - -// ... - -if (Astro.request.method === 'POST') { - - // Fetch the Xata instance - const xata = getXataClient() - - // ... - - // Create a blur URL of the user image - - // Using Xata image transformations to obtain the image URL - // with a fixed height and width and 75% of it blurred - const userBlurURL = transformImage(userImageURL, { - blur: 75, - width: 100, - height: 100, - }) - - // Create a Base64 hash of the blur image URL - const userBlurHash = await createBlurHash(userBlurURL) - - // Create the user record with the slug - - // Redirect user to the next step -} ---- -``` - -## Syncing profiles using the Xata SDK - -After you have generated blurred images, the last step in publishing profiles (similar to whatā€™s done in publishing photographs) is to create a user record with relevant details using the Xata TypeScript SDKā€™s `create` command. In Astro, we then set the successful message for the user before they are redirected to the photograph upload page. The integrated conditional rendering ensures a visual cue of the operation's success or failure, providing a responsive and user-friendly experience. - -```astro ---- -// File: src/pages/profile/create.astro - -// Import the Xata Client created by Xata CLI in src/xata.ts -import { getXataClient } from '@/xata' - -const response = { form: false, message: '', created: false, redirect: null } - -// ... - -if (Astro.request.method === 'POST') { - - // ... - - // Create the user record with the slug - await xata.db.profiles.create({ - slug: userSlug, - name: userName, - image: userImage, - width: userImageW, - height: userImageH, - imageHash: userBlurHash, - }) - - // Send the user to photograph upload page - response.redirect = '/photograph/create' - - // Set the relevant message for the user - response.message = 'Published profile successfully. Redirecting you to upload your first photograph...' -} ---- - - -{ - response.form && - (response.created ? ( -

{response.message}

- ) : ( -

{response.message}

- )) -} - - - - - -``` - -## Using Xata query - -The user profile page (`src/pages/[profile]/index.astro`) leverages the Xata Client to dynamically fetch and display all the photographs (not paginated) for a specific user profile. To engage users visually as soon as they open the gallery, we use the stored `imageHash` value as the background of the images (to be loaded). To prevent CLS, we use the stored `width` and `height` values to inform the browser of the expected dimensions of the images. - -```astro ---- -// File: src/pages/[profile]/index.astro - -// Import the Xata Client created by Xata CLI in src/xata.ts -import { getXataClient } from '@/xata' - -import Layout from '@/layouts/Layout.astro' - -// Get the profile slug from url path -const { profile } = Astro.params - -// Fetch the Xata instance -const xata = getXataClient() - -// Get all the photographs related to the profile -const profilePhotographs = await xata.db.photographs - // Filter the results to the specific profile - .filter({ 'profile-slug': profile }) - // Get all the photographs - .getAll() ---- - - -
- { - profilePhotographs.map( - ({ width: photoW, height: photoH, name: photoName, image: photoImageURL, tagline: photoTagline, imageHash: photoImageHash }, _) => ( - {/* Destructure the width and height to prevent CLS */} - {photoName} - ), - ) - } -
- -``` - -![(Image gallery - presenting blur images initially)](/images/build-image-gallery/image14.png) - -![(Image gallery - after image loads complete)](/images/build-image-gallery/image15.png) - -## Deploy to Cloudflare Pages - -The repository is ready to deploy to Cloudflare. Follow the steps below to deploy seamlessly with Cloudflare šŸ‘‡šŸ» - -1. Create a GitHub repository with the app code. -1. Click **Create application** in the Workers & Pages section of Cloudflare dashboard. -1. Navigate to the **Pages** tab and select **Connect to Git**. -1. Link the created GitHub repository as your new project. -1. Scroll down and update the **Framework preset** to **Astro**. -1. Update the environment variables from the `.env` locally. -1. Click **Save and Deploy** and go back to the project **Settings** > **Functions**. -1. Add `nodejs_compat` to the **Compatibility flags** section -1. Deploy! šŸš€ - -### Why Cloudflare Pages? - -Cloudflare Pages stood out for this particular use case as it [offers up to 100 MB request body size in their Free plan](https://developers.cloudflare.com/workers/platform/limits/#request-limits). This helped to bypass the 4.5MB request body size limit in various serverless hosting providers. - -## More information - -For more detailed insights, explore the references cited in this post. - -| Resource | Link | -| --------------------------- | ------------------------------------------------------------------------------------------------ | -| Demo Image Gallery | [Demo](https://image-gallery-cms-with-astro-xata-cloudflare.pages.dev/rishi) | -| GitHub Repo | [Repository](https://github.com/rishi-raj-jain/image-gallery-cms-with-astro-xata-cloudflare) | -| Astro with Xata | [Getting Started Guide](https://xata.io/docs/getting-started/astro) | -| Astro View Transition Forms | [Astro Transitions](https://docs.astro.build/en/guides/view-transitions/#transitions-with-forms) | -| Xata File Attachments | [Xata File APIs](https://xata.io/docs/sdk/file-attachments#upload-a-file-using-file-apis) | -| Xata Transformations | [Xata Image Transformations](https://xata.io/docs/sdk/image-transformations) | -| Xata Get Records | [Xata Querying](https://xata.io/docs/sdk/get#the-typescript-sdk-functions-for-querying) | -| Cloudflare Workers Limits | [Cloudflare Limits](https://developers.cloudflare.com/workers/platform/limits/#request-limits) | - -## What's next? - -We'd love to hear from you if you have any feedback on this tutorial, would like to know more about Xata, or if you'd like to contribute a community blog or tutorial. Reach out to us on [Discord](https://discord.com/invite/kvAcQKh7vm) or join us on [X | Twitter](https://twitter.com/xata). Happy building šŸ¦‹ diff --git a/building-protected-and-paywall.mdx b/building-protected-and-paywall.mdx deleted file mode 100644 index 98b0fc4f..00000000 --- a/building-protected-and-paywall.mdx +++ /dev/null @@ -1,969 +0,0 @@ ---- -title: 'Build authenticated and paywall pages with Stripe and Xata' -description: 'Learn how to build protected and paywall pages using using Astro, Stripe, Lucia Auth, Xata and Vercel.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/building-protected-and-paywall/authenticated-paywall-stripe-xata.png - alt: How to build paywall pages with Stripe, Lucia Auth and Xata -author: Rishi Raj Jain -date: 04-10-2024 -published: true -slug: authenticated-paywall-stripe-xata -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/building-protected-and-paywall/authenticated-paywall-stripe-xata.png ---- - -In this post, you'll learn how to authenticate users using Xata, create Stripe checkouts and respond to Stripe webhooks in an Astro application. - -- Set up Xata -- Create a schema with different column types -- Programmatically create Stripe checkout sessions -- Respond to Stripe webhooks -- Authenticate users with Lucia Auth - -## Before you begin - -### Prerequisites - -You'll need the following: - -- [Node.js 18](https://nodejs.org/en/blog/announcements/v18-release-announce) or later -- [pnpm](https://pnpm.io/installation) package manager -- A [Xata](https://xata.io/) account -- A [Stripe](https://stripe.com) account - -### Tech Stack - -The following technologies are used in this guide: - -| Technology | Description | -| ------------------------------------- | ---------------------------------------------------------------------------------- | -| [Xata](https://xata.io/) | Serverless Postgres database platform for scalable, real-time applications. | -| [Astro](https://astro.build/) | Framework for building fast, modern websites with serverless backend support. | -| [Stripe](https://stripe.com) | A platform to accept payments globally. | -| [Lucia Auth](https://lucia-auth.com/) | An authentication library that abstracts away the complexity of handling sessions. | - -# Steps - -To complete this guide, you will need to follow these steps: - -- [Create a new Astro application](#create-a-new-astro-application) -- [Add Tailwind CSS to the application](#add-tailwind-css-to-the-application) -- [Enable Server Side Rendering in Astro](#enabling-server-side-rendering-in-astro) -- [Set up a Xata Database](#setting-up-a-xata-database) - - [Install the Xata CLI](#install-the-xata-cli) -- [Create Stripe Checkout Sessions](#create-stripe-checkout-sessions) -- [Respond to Stripe Webhooks](#respond-to-stripe-webhooks) -- [Authenticate users with Lucia Auth and Xata](#authenticating-users-with-lucia-auth-and-xata) - - [Build a Xata Adapter for Lucia Auth](#building-xata-adapter-for-lucia-auth) - - [Initialize Lucia with Xata](#initialize-lucia-with-xata) - - [Create Utilities for User Authentication Status](#create-utilities-for-user-authentication-status) - - [Build Authentication Routes with Lucia Auth](#building-authentication-routes-with-lucia-auth) - - [Build the Sign Up HTML and API Route](#build-the-sign-up-html-and-api-route) - - [Build the Sign In HTML and API Route](#build-the-sign-in-html-and-api-route) - - [Build the Sign Out API Route](#build-the-sign-out-api-route) - - [Create Protected and Paid Routes](#create-protected-and-paid-routes) -- [Deploy to Vercel](#deploy-to-vercel) - -## Create a new Astro application - -Letā€™s get started by creating a new Astro project. Open your terminal and run the following command: - -```bash -pnpm create astro@latest protected-and-paid-stripe-xata -``` - -`pnpm create astro` is the recommended way to scaffold an Astro project quickly. - -When prompted, choose: - -- `Empty` when prompted on how to start the new project. -- `Yes` when prompted if you plan to write using Typescript. -- `Strict` when prompted how strict Typescript should be. -- `Yes` when prompted to install dependencies. -- `Yes` when prompted to initialize a git repository. - -Once thatā€™s done, you can move into the project directory and start the app: - -```bash -cd protected-and-paid-stripe-xata -pnpm dev -``` - -The app should be running on [localhost:4321](http://localhost:4321/). Now, let's close the development server as we move on to integrating Tailwind CSS in the application. - -## Add Tailwind CSS to the application - -For styling the app, you will use Tailwind CSS. Install and set up Tailwind at the root of your project's directory by running: - -```bash -pnpm astro add tailwind -``` - -When prompted, choose: - -- `Yes` when prompted to install the Tailwind dependencies. -- `Yes` when prompted to generate a minimal `tailwind.config.mjs` file. -- `Yes` when prompted to make changes to the Astro configuration file. - -The command then finishes integrating TailwindCSS into your Astro project. It installed the following dependency: - -- `tailwindcss`: TailwindCSS as a package to scan your project files to generate corresponding styles. -- `@astrojs/tailwind`: The adapter that brings Tailwind's utility CSS classes to every `.astro` file and framework component in your project. - -Now, let's move on to enabling server side rendering in the application. - -## Enabling Server Side Rendering in Astro - -To create checkouts using the Stripe API dynamically, you will need to server-side render your Astro application. Enable server side rendering in your Astro project by executing the following command in your terminal: - -```bash -pnpm add @astrojs/vercel -``` - -When prompted, choose: - -- `Yes` when prompted to install the Vercel dependencies. -- `Yes` when prompted to make changes to the Astro configuration file. - -The command above installed the following dependency: - -- `@astrojs/vercel`: The adapter that allows you to server-side render your Astro application on Vercel. - -Now, let's move on to integrating Xata in the application. - -## Setting up a Xata Database - -After you've created a Xata account and are logged in, create a database by clicking on `+ Add database`. - -![Create Xata database](/images/building-protected-and-paywall/create-database.png) - -### Install the Xata CLI - -Obtain the Xata initialization command by clicking on a table (say, `user`), and then on the `Get code snippet` button. - -![View User Table](/images/building-protected-and-paywall/user-table.png) - -A modal titled `Code snippets` is presented which contains a set of commands to integrate Xata into your project, locally. - -![Code Snippet for Xata CLI](/images/building-protected-and-paywall/code-snippet.png) - -For interacting with your Xata database using the TypeScript SDK, install the Xata CLI globally by executing the following command in your terminal: - -```bash -# Installs the CLI globally -npm install -g @xata.io/cli -``` - -Then, link your Xata project to your Astro application by executing the following command in your terminal window: - -```bash -# Initialize your project locally with the Xata CLI -xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.us-east-1.xata.sh/db/astro-stripe-protected-and-paid -``` - -Use the following answers to the Xata CLI one-time setup question prompts to integrate Xata with Astro: - -- `Yes` when prompted to add `.env` to `.gitignore`. -- `src/xata.ts` when prompted to enter the output path for the generated code by the Xata CLI. - -![Setup Astro project with Xata CLI](/images/building-protected-and-paywall/cli.png) - -Now, letā€™s move onto creating API endpoints that create checkouts dynamically using Stripe. - -## Create Stripe Checkout Sessions - -A Stripe checkout session, in this case, would refer to a payment link hosted by Stripe where users can pay for a given product. - -Let's use the index route to display a buy button that would take users to the Stripe hosted checkout. Create an index route (`src/pages/index.astro`) with the following code: - -```astro ---- - -// File: src/pages/index.astro - -## export const prerender = true - - - - - - - -
- -
- - -``` - -The code above creates an index route in the Astro application containing a form that POSTs to `/api/stripe/checkout`. To handle the form submission, create a file `src/pages/api/stripe/checkout.ts` with the following code: - -```tsx -// File: src/pages/api/stripe/checkout.ts - -import Stripe from 'stripe'; -import type { APIContext } from 'astro'; - -export async function POST({ redirect }: APIContext) { - const STRIPE_SECRET_KEY = import.meta.env.STRIPE_SECRET_KEY; - if (!STRIPE_SECRET_KEY) return new Response(null, { status: 500 }); - const stripe = new Stripe(STRIPE_SECRET_KEY, { apiVersion: '2023-10-16' }); - const session = await stripe.checkout.sessions.create({ - mode: 'payment', - payment_method_configuration: 'pmc_1O2qH3SE9voLRYpuz5FLmkvn', - line_items: [ - { - quantity: 1, - price_data: { - unit_amount: 0, - currency: 'usd', - product: 'prod_OqWkk7Rz9Yw18f' - } - } - ], - success_url: 'http://localhost:4321' - }); - return redirect(session.url); -} -``` - -The code above does the following: - -- Imports the Stripe SDK and the type APIContext by Astro. -- Exports a `POST` HTTP request handler. -- Validates the presence of `STRIPE_SECRET_KEY` environment variable. -- Creates a new stripe object using the secret key and the latest api version. -- Creates a checkout with a given item adjusted to be for free. Having a checkout containing free items will help you quickly test a purchase flow manually. -- Redirects user to the newly created checkout's URL. - -Let's move on to handling the webhook requests triggered by Stripe if there's a successful purchase on the payment link. - -## Respond to Stripe Webhooks - -Upon a successful purchase, Stripe fires a webhook POST request. To handle the webhook requests in Astro via server-side code, create a `src/pages/api/stripe/webhook.ts` file with the following code: - -```tsx -// File: src/pages/api/stripe/webhook.ts - -import Stripe from 'stripe'; -import { getXataClient } from '@/xata'; -import type { APIContext } from 'astro'; - -// Process rawBody from the request Object -async function getRawBody(request: Request) { - let chunks = []; - let done = false; - const reader = request.body.getReader(); - while (!done) { - const { value, done: isDone } = await reader.read(); - if (value) { - chunks.push(value); - } - done = isDone; - } - const bodyData = new Uint8Array(chunks.reduce((acc, chunk) => acc + chunk.length, 0)); - let offset = 0; - for (const chunk of chunks) { - bodyData.set(chunk, offset); - offset += chunk.length; - } - return Buffer.from(bodyData); -} - -// Stripe API Reference -// https://stripe.com/docs/webhooks#webhook-endpoint-def -export async function POST({ request }: APIContext) { - try { - const STRIPE_SECRET_KEY = import.meta.env.STRIPE_SECRET_KEY; - const STRIPE_WEBHOOK_SIG = import.meta.env.STRIPE_WEBHOOK_SIG; - if (!STRIPE_SECRET_KEY || !STRIPE_WEBHOOK_SIG) return new Response(null, { status: 500 }); - const stripe = new Stripe(STRIPE_SECRET_KEY, { apiVersion: '2023-10-16' }); - const rawBody = await getRawBody(request); - let event = JSON.parse(rawBody.toString()); - const sig = request.headers.get('stripe-signature'); - try { - event = stripe.webhooks.constructEvent(rawBody, sig, STRIPE_WEBHOOK_SIG); - } catch (err) { - console.log(err.message); - return new Response(`Webhook Error: ${err.message}`, { status: 400 }); - } - if (event.type === 'checkout.session.completed' || event.type === 'payment_intent.succeeded') { - const email = event.data.object?.customer_details?.email; - if (email) { - const xata = getXataClient(); - const existingRecord = await xata.db.user.filter({ email }).getFirst(); - if (existingRecord) { - await xata.db.user.update(existingRecord.id, { paid: true }); - } else { - await xata.db.user.create({ email, paid: true }); - } - return new Response('marked the user as paid', { status: 200 }); - } - return new Response('no email of the user is found', { status: 200 }); - } - return new Response(JSON.stringify(event), { status: 404 }); - } catch (e) { - return new Response(e.message || e.toString(), { status: 500 }); - } -} -``` - -The code above does the following: - -- Imports the Stripe class from its SDK, `APIContext` type by Astro and `getXataClient` to access the in-memory Xata instance. -- Exports a POST HTTP handler to only respond to incoming POST request(s). -- Validates the presence of `STRIPE_SECRET_KEY` and `STRIPE_WEBHOOK_SIG` environment variables. -- Creates a Stripe webhook event from the incoming Stripe webhook request body. -- Checks for an existing record with the email entered during the checkout. -- If a record is found, the `paid` attribute is set to `true`. -- Otherwise, it creates a record with the email and `paid` attribute as `true`. - -Let's move on to authenticating users with your Xata database and a Lucia library. - -## Authenticating users with Lucia Auth and Xata - -To start authenticating users and managing their sessions, install Lucia and Oslo, for various authentication utilities by executing the following command in your terminal: - -```bash -pnpm add lucia oslo dotenv -``` - -The above command installs the packages passed to the `install` command. The libraries installed include: - -- `lucia`: An open source authentication library that abstracts away the complexity of handling sessions. -- `dotenv`: A library for handling environment variables. -- `oslo`: A collection of authentication related utilities. - -```diff -// File: astro.config.mjs - -+ import 'dotenv/config' -import node from '@astrojs/node' -import tailwind from '@astrojs/tailwind' -import { defineConfig } from 'astro/config' - -export default defineConfig({ - output: 'server', - adapter: node({ - mode: 'standalone', - }), - integrations: [tailwind()], -+ vite: { -+ optimizeDeps: { -+ exclude: ['oslo'], -+ }, -+ }, -}) -``` - -The code additions above do the following: - -- Imports dotenv which makes all the environment variables accessible via the `process.env` object. -- Uses vite's [optimizeDeps](https://vitejs.dev/config/dep-optimization-options#optimizedeps-exclude) to exclude the `oslo` library from being pre-bundled, effectively, using the library as is. - -Now, create a Lucia directory in the src directory by running the following command: - -```bash -mkdir src/lucia -``` - -Let's now move on to creating a Xata adapter for our Lucia library. - -### Building a Xata Adapter for Lucia Auth - -Create a `src/lucia/xata.ts` file with the following code: - -```tsx -// File: src/lucia/xata.ts - -import { getXataClient } from '@/xata'; -import type { - Adapter, - DatabaseSession, - RegisteredDatabaseSessionAttributes, - DatabaseUser, - RegisteredDatabaseUserAttributes, - UserId -} from 'lucia'; -``` - -The code above imports the `getXataClient` utility to retrieve the in-memory Xata instance. It then imports the relevant types from the `lucia` module. Append the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -interface UserSchema extends RegisteredDatabaseUserAttributes { - id: string; -} - -interface SessionSchema extends RegisteredDatabaseSessionAttributes { - id: string; - user_id: string; - expires_at: Date; -} -``` - -The code above defines the following types: - -- `UserSchema`: A type created with `RegisteredDatabaseUserAttributes` type definition along with `id` attribute as a string. -- `SessionSchema`: A type created with `RegisteredDatabaseSessionAttributes` type definition along with `id` attribute as a string, `user_id` attribute as a string and `expires_at` attribute as a Date. - -Append the following code to handle data transformations to Lucia's expected structure: - -```tsx -// File: src/lucia/xata.ts - -// ... - -function transformIntoDatabaseSession(raw: SessionSchema): DatabaseSession { - const { id, user_id: userId, expires_at: expiresAt, ...attributes } = raw; - return { - id, - userId, - expiresAt, - attributes - }; -} - -function transformIntoDatabaseUser(raw: UserSchema): DatabaseUser { - const { id, ...attributes } = raw; - return { - id, - attributes - }; -} -``` - -The code above defines the following methods: - -- `transformIntoDatabaseSession`: returns the `SessionSchema` object in the shape of `DatabaseSession` type. -- `transformIntoDatabaseUser`: returns the `UserSchema` object in the shape of `DatabaseUser` type. - -Append the following code to create the XataAdapter class: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - private controller = getXataClient(); - - // ... -} -``` - -The code above defines a class XataAdapter based on Lucia's Adapter class. Then, it sets a `controller` private class variable equivalent to the Xata instance obtained from the `getXataClient` method. Define a method to delete a session by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async deleteSession(sessionId: string): Promise { - await this.controller.sql`DELETE FROM "session" WHERE id=${sessionId}`; - } -} -``` - -The code above defines a `deleteSession` method that uses Xata SQL queries to delete a particular session from your database. Define a method to delete all the sessions for a user by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async deleteUserSessions(userId: UserId): Promise { - await this.controller.sql`DELETE FROM "session" WHERE user_id=${userId}`; - } -} -``` - -The code above defines a `deleteUserSessions` method that uses Xata SQL queries to delete all sessions pertaining to a particular user in your database. Define a method to retrieve all the sessions for a user by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async getUserSessions(user_id: UserId): Promise { - const records = await this.controller.db.session.filter({ user_id }).getAll(); - return records.map((val) => { - return transformIntoDatabaseSession(val); - }); - } -} -``` - -The code above defines a `getUserSessions` method that uses Xata SDK methods to retrieve all sessions pertaining to a particular user in your database. Define a method to create a session by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async setSession(databaseSession: DatabaseSession): Promise { - await this.controller.db.session.create({ - id: databaseSession.id, - user_id: databaseSession.userId, - expires_at: databaseSession.expiresAt, - ...databaseSession.attributes - }); - } -} -``` - -The code above defines a `setSession` method that uses Xata SDK methods to insert a record in a session table in your database. Define a method to update a session by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async updateSessionExpiration(sessionId: string, expiresAt: Date): Promise { - await this.controller.sql`UPDATE "session" SET expires_at=${expiresAt} WHERE id=${sessionId}`; - } -} -``` - -The code above defines an `updateSessionExpiration` method that uses a Xata SQL query to update the `expires_at` attribute in a particular session in your database. Define a method to delete all expired sessions by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async deleteExpiredSessions(): Promise { - await this.controller.sql`DELETE FROM "session" WHERE expires_at <=${new Date()}`; - } -} -``` - -The code above defines a `deleteExpiredSessions` method that uses a Xata SQL query to delete all the sessions whose `expires_at` attribute have expired. Define a method to obtain a session by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - private async getSession(session_id: string): Promise { - const records = await this.controller.db.session.filter({ session_id }).getFirst(); - return transformIntoDatabaseSession(records); - } -} -``` - -The code above defines a `getSession` method that uses Xata SDK methods to retrieve a single matching record for a given session from your database. Define a method to obtain a user associated with a session by appending the following code: - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - private async getUserFromSessionId(session_id: string): Promise { - const theSession = await this.controller.db.session.filter({ session_id }).getFirst(); - if (theSession) { - const { user_id } = theSession; - const getUser = await this.controller.db.user.filter({ user_id }).getFirst(); - if (getUser) return transformIntoDatabaseUser(getUser); - } - return null; - } -} -``` - -The code above defines a `getUserFromSessionId` method that uses Xata SDK methods to retrieve a single matching record for a given session from your database. If a session is found, it further queries to obtain the user associated with the session. Otherwise, it returns `null`. - -```tsx -// File: src/lucia/xata.ts - -// ... - -export class XataAdapter implements Adapter { - // ... - - public async getSessionAndUser( - sessionId: string - ): Promise<[session: DatabaseSession | null, user: DatabaseUser | null]> { - const [databaseSession, databaseUser] = await Promise.all([ - this.getSession(sessionId), - this.getUserFromSessionId(sessionId) - ]); - return [databaseSession, databaseUser]; - } -} -``` - -The code above defines a `getSessionAndUser` method that uses the `getSession` and `getUserFromSessionId` methods to obtain both the user and session information associated with the particular session id. - -The XataAdapter class is now ready to be used as Lucia's adapter. - -### Initialize Lucia with Xata - -To use Xata with Lucia, create a file `index.ts` inside the `lucia` directory with the following code: - -```tsx -// File: src/lucia/index.ts - -import { Lucia } from 'lucia'; -import { XataAdapter } from './xata'; - -const adapter = new XataAdapter(); - -export const lucia = new Lucia(adapter, { - sessionCookie: { - attributes: { - secure: import.meta.env.PROD - } - }, - getUserAttributes: (attributes) => { - return { - paid: attributes.paid, - email: attributes.email - }; - } -}); -``` - -The code above does the following: - -- Imports the Lucia and XataAdapter classes. -- Creates a new instance of XataAdapter as `adapter`. -- Uses the `adapter` to create a new Lucia instance wherein the session cookie is set to `Secure` if the application is running in the production environment. -- The `lucia` instance defines the `getUserAttributes` function to only fetch `paid` and `email` attributes from the user record(s). - -To create type definitions for the associated user and session, append the following code: - -```tsx -// File: src/lucia/index.ts - -// ... - -interface DatabaseUserAttributes { - email: string; - paid: boolean | null; -} - -declare module 'lucia' { - interface Register { - Lucia: typeof lucia; - DatabaseUserAttributes: DatabaseUserAttributes; - } -} -``` - -The code above defines the following types: - -- `DatabaseUserAttributes`: An object that would contain an `email` attribute as a `string`, and `paid` attribute as `boolean` or `null`. -- `Lucia`: As the type of the lucia instance created. - -### Create Utilities for User Authentication Status - -To retrieve the authentication status of the session, you will need to obtain the user information associated with the request. Create a file `user.ts` with the following code: - -```tsx -// File: src/lucia/user.ts - -import { lucia } from '.'; -import type { User } from 'lucia'; -import type { AstroCookies } from 'astro'; - -export function getSessionID(cookies: AstroCookies): string | null { - const auth_session = cookies.get('auth_session'); - if (!auth_session) return null; - return lucia.readSessionCookie(`auth_session=${auth_session.value}`); -} - -export async function getUser(cookies: AstroCookies): Promise { - const session_id = getSessionID(cookies); - if (!session_id) return null; - const { user } = await lucia.validateSession(session_id); - return user; -} -``` - -The code above does the following: - -- Imports the `lucia` instance and the `User` type by Lucia. -- Exports a `getSessionID` function which uses Lucia's `readSessionCookie` utility to obtain the session ID. -- Exports a `getUser` function which uses the `getSessionID` function to obtain the session id. Then, it uses Lucia's `validateSession` utility to obtain the user data information with the session. - -Let's move on to creating simple user authentication forms and the API routes to handle the authentication logic. - -### Building Authentication Routes with Lucia Auth - -In this section, you will learn how to create simple forms in Astro, and use the form data in Astro endpoints to sign up, sign out and sign in the users in your application. - -#### Build the Sign Up HTML and API Route - -To serve responses to a `/signup` route in Astro, create a `src/pages/signup.astro` file with the following code: - -```astro ---- -// File: src/pages/signup.astro - -export const prerender = true ---- - - - - - - - -

Sign Up

-
- - - - - -
- - -``` - -The code above creates a sign up route in the Astro application containing a form that POSTs to `/api/sign/up` with the user's email and password. Create a `src/pages/api/sign/up.ts` file with the following code to handle the form submission: - -```tsx -// File: src/pages/api/sign/up.ts - -import { generateId } from 'lucia'; -import { lucia } from '@/lucia/index'; -import { getXataClient } from '@/xata'; -import type { APIContext } from 'astro'; -import { Argon2id } from 'oslo/password'; - -export async function POST({ request, redirect, cookies }: APIContext): Promise { - const xata = getXataClient(); - const user_id = generateId(15); - const formData = await request.formData(); - const email = (formData.get('email') as string).trim(); - const password = formData.get('password') as string; - const hashed_password = await new Argon2id().hash(password); - const existingRecord = await xata.db.user.filter({ email }).getFirst(); - if (existingRecord) { - if (existingRecord.hashed_password !== null) return redirect('/signin'); - await xata.db.user.update(existingRecord.id, { user_id, hashed_password }); - } else { - await xata.db.user.create({ email, user_id, hashed_password }); - } - const session = await lucia.createSession(user_id, {}); - const sessionCookie = lucia.createSessionCookie(session.id); - cookies.set(sessionCookie.name, sessionCookie.value, sessionCookie.attributes); - return redirect('/'); -} -``` - -The code above does the following: - -- Imports a utility from the Lucia SDK to create random ID(s), the `getXataClient` utility to obtain the in-memory Xata instance, Argon2id class by `oslo` to hash given password(s). -- Exports a POST HTTP handler to only respond to incoming POST request(s). -- Obtains the email and password from the request's form data. -- Hashes the password using Argon2id's `hash` method. -- Retrieves a single matching record using the email from form data. -- If no such record is found, it creates a record in the user table with the email, user id and hashed password. -- Otherwise, if the hashed password is present in the matching record, it redirects to `/signin` route. Otherwise, it updates the existing record with the newly signed up user. -- Creates a session using `createSession` helper utility by Lucia and sets the cookie. -- Redirects to the index route. - -#### Build the Sign In HTML and API Route - -To serve responses to a `/signin` route in Astro, create a `src/pages/signin.astro` file with the following code: - -```astro ---- -// File: src/pages/signin.astro - -export const prerender = true ---- - - - - - - - -

Sign In

-
- - - - - -
- - -``` - -The code above creates a sign in route in the Astro application containing a form that POSTs to `/api/sign/in` with the user's email and password. Create a `src/pages/api/sign/in.ts` file with the following code to handle the form submission: - -```tsx -// File: src/pages/api/sign/in.ts - -import { lucia } from '@/lucia/index'; -import { getXataClient } from '@/xata'; -import type { APIContext } from 'astro'; -import { Argon2id } from 'oslo/password'; - -export async function POST({ request, cookies, redirect }: APIContext): Promise { - const xata = getXataClient(); - const formData = await request.formData(); - const email = (formData.get('email') as string).trim(); - const password = formData.get('password') as string; - const existingRecord = await xata.db.user.filter({ email }).getFirst(); - if (!existingRecord) return new Response('Incorrect email or password', { status: 400 }); - const validPassword = await new Argon2id().verify(existingRecord.hashed_password, password); - if (!validPassword) return new Response('Incorrect email or password', { status: 400 }); - const session = await lucia.createSession(existingRecord.user_id, {}); - const sessionCookie = lucia.createSessionCookie(session.id); - cookies.set(sessionCookie.name, sessionCookie.value, sessionCookie.attributes); - return redirect('/'); -} -``` - -The code above does the following: - -- Imports the initialized `lucia` instance and `getXataClient` helper utility. -- Exports a POST HTTP handler to only respond to incoming POST request(s). -- Obtains the email and password from the request's form data. -- Looks for an existing record with the email in user table of your database. -- If not found, it returns a 400 Bad Request response. -- Otherwise, it uses `Argon2id` to verify the hashed password. -- If the hashed password does not match, it returns a 400 Bad Request response. -- Otherwise, it creates a session using `createSession` helper utility by Lucia and sets the cookie. -- Redirects to the index route. - -#### Build the Sign Out API Route - -To sign out users in your Astro application, create a `src/pages/api/sign/out.ts` file with the following code: - -```tsx -// File: src/pages/api/sign/out.ts - -import { lucia } from '@/lucia/index'; -import type { APIContext } from 'astro'; -import { getSessionID } from '@/lucia/user'; - -export async function GET({ cookies, redirect }: APIContext): Promise { - await lucia.invalidateSession(getSessionID(cookies)); - const sessionCookie = lucia.createBlankSessionCookie(); - cookies.set(sessionCookie.name, sessionCookie.value, sessionCookie.attributes); - return redirect('/'); -} -``` - -The code above does the following: - -- Imports the initialized `lucia` instance and `getSessionID` helper utility. -- Exports a GET HTTP handler to only respond to incoming GET request(s). -- Invalidates the current user session using the `invalidateSession` utility by Lucia. -- Creates an empty user session cookie using `createBlankSessionCookie` utility by Lucia. -- Sets the blank user session cookie. -- Redirects to the index route. - -#### Create Protected and Paid Routes - -To restrict access to a page for only paid and authenticated users, you will obtain the user information using the utilities created earlier. Create a file `src/pages/protected_and_paid.astro` with the following code: - -```astro ---- -// File: src/pages/protected_and_paid.astro - -import { getUser } from '@/lucia/user' - -const user = await getUser(Astro.cookies) - -if (!user || user.paid !== true) return new Response(null, { status: 403 }) ---- - - - - - - - - Protected and Paid Content - - -``` - -The code above creates a `/protected_and_paid` route in the Astro application that does the following: - -- Checks if there's a user associated with the session and if they have paid. -- If the above is not true, it returns a 403 Unauthorized response. Otherwise, it returns an HTML page containing the paid and protected information. - -To restrict access to a page for only authenticated users, you will obtain the user information using the utilities created earlier. Create a file `src/pages/protected.astro` with the following code: - -```astro ---- -import { getUser } from '@/lucia/user' - -const user = await getUser(Astro.cookies) - -if (!user) return new Response(null, { status: 403 }) ---- - - - - - - - - Protected Content - - -``` - -The code above creates a `/protected` route in the Astro application that does the following: - -- Checks if there's a user associated with the session. -- If the above is not true, it returns a 403 Unauthorized response. Otherwise, it returns an HTML page containing the protected information. - -## Deploy to Vercel - -The repository, is now ready to deploy to Vercel. Use the following steps to deploy: - -- Start by creating a GitHub repository containing your app's code. -- Then, navigate to the Vercel Dashboard and create a **New Project**. -- Link the new project to the GitHub repository you've just created. -- In **Settings**, update the **Environment Variables** to match those in your local `.env` file. -- Deploy! šŸš€ - -## More Information - -For more detailed insights, explore the references cited in this post. - -- [GitHub Repo](https://github.com/rishi-raj-jain/protected-and-paid-stripe-xata) -- [Adapter - Lucia Auth](https://lucia-auth.com/reference/main/Adapter) -- [Credentials Lucia Auth in Astro](https://lucia-auth.com/tutorials/username-and-password/astro) -- [Create a Stripe Checkout Session](https://docs.stripe.com/api/checkout/sessions/create) - -## Whatā€™s next? - -We'd love to hear from you if you have any feedback on this tutorial, would like to know more about Xata, or if you would like to contribute a community blog or tutorial. Reach out to us on [Discord](https://discord.com/invite/kvAcQKh7vm) or join us on [X | Twitter](https://twitter.com/xata). Happy building šŸ¦‹ diff --git a/cache-openai-langchain.mdx b/cache-openai-langchain.mdx deleted file mode 100644 index 098c2472..00000000 --- a/cache-openai-langchain.mdx +++ /dev/null @@ -1,517 +0,0 @@ ---- -title: 'Caching OpenAI Chat API Responses with LangChain and Xata' -description: 'Learn how to cache OpenAI Chat API responses in LangChain with Xata.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/cache-openai-langchain/cache-openai-langchain-xata.jpg - alt: Xata -author: Rishi Raj Jain -date: 03-27-2024 -published: true -slug: cache-openai-langchain-xata -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/cache-openai-langchain/cache-openai-langchain-xata.jpg ---- - -In this guide, you'll learn how to cache OpenAI Chat API responses in different ways using LangChain and Xata. Youā€™ll learn how to: - -- Set up a Xata Database -- Cache LangChain ChatOpenAI Responses using Callbacks -- Cache LangChain ChatOpenAI Responses using Cache Layer - -## Before you begin - -### Prerequisites - -You'll need the following: - -- [Node.js 18](https://nodejs.org/en) or later -- [pnpm](https://pnpm.io/installation) package manager -- A [Xata](https://app.xata.io/signin?mode=signup) account -- An [OpenAI](https://platform.openai.com) account - -### Tech Stack - -The following technologies are used in this guide: - -| Technology | Description | -| -------------------------------------- | ------------------------------------------------------------------------------------------------- | -| [Express.js](https://expressjs.com/) | Fast, unopinionated, minimalist web framework for Node.js. | -| [Xata](https://xata.io/) | Serverless database platform for scalable, real-time applications. | -| [OpenAI](https://openai.com/) | OpenAI is an artificial intelligence research lab focused on developing advanced AI technologies. | -| [LangChain](https://js.langchain.com/) | Framework for developing applications powered by language models. | - -## Why Cache OpenAI API Responses? - -Caching responses from the OpenAI API significantly speeds up your application and save costs by reducing the number of unnecessary API calls. This is particularly useful for applications that generate similar queries often, as responses can be stored and reused. By integrating LangChain and Xata, you can efficiently cache responses in a scalable and real-time manner. - -## Create a new Express.js application - -To quickly create endpoints demonstrating different ways of caching OpenAI Chat API responses, youā€™re going to create an Express.js application. To start building the application, create a new directory for your Express.js project. Open your terminal and run the following command: - -```bash -mkdir xata-cache-openai-langchain -cd xata-cache-openai-langchain -``` - -Then, create a blank `npm` project by executing the following command: - -```bash -npm init -y -``` - -Next, in your first terminal window, run the command below to install the necessary libraries and packages for building the application: - -```bash -pnpm add express -pnpm add -D dotenv -``` - -The above command installs the packages passed to the `install` command, with the `-D` flag specifying the libraries intended for development purposes only. - -The libraries installed include: - -- `express`: A minimalist web framework for Node.js. - -The development-specific libraries include: - -- `dotenv`: A library for handling environment variables. - -Finally, create an `index.js` file with the following code: - -```tsx -// File: index.js - -require('dotenv/config'); - -const PORT = 3005; - -const express = require('express'); -const app = express(); - -app.use(express.json()); - -app.get('/', (req, res) => { - res.send('Home'); -}); - -app.listen(PORT, () => { - console.log(`Listening on http://localhost:${PORT}`); -}); -``` - -The above code loads all the environments variables into the scope using dotenvā€™s config method, making them accessible via the `process.env` object. Further, it uses Expressā€™s json middleware to serialize incoming JSON POST request bodies. Finally, it sets up the Expressā€™s server to listen to incoming requests at port `3005`. - -To start running the application, execute the following command in your terminal: - -```bash -node index.js -``` - -The app should be running on [localhost:3005](http://localhost:3005). Currently, it just displays `Home` on the `/` page. - -## Setting up a Xata Database - -After you've created a Xata account and are logged in, create a database by clicking on `+ Add database`. - -![Create a Xata Database](/images/cache-openai-langchain/Screenshot_2024-03-07_at_11.25.35_PM.png) - -Then, create two tables, named `responses` and `invocations` that will maintain the cache of the queries and the responses generated by OpenAI APIs. - -![Create a table](/images/cache-openai-langchain/Screenshot_2024-03-07_at_11.43.35_PM.png) - -Now, letā€™s move on to integrating Xata in your Express.js application. - -### Install the Xata CLI - -Obtain the Xata initialization command by clicking on a table (say, `responses`), and then on `Get code snippet` button. - -![Browse newly initialized table](/images/cache-openai-langchain/Screenshot_2024-03-07_at_11.47.45_PM.png) - -A modal titled `Code snippets` is presented which contains a set of commands to integrate Xata into your project, locally. - -![Obtain Xata Setup Commands](/images/cache-openai-langchain/Screenshot_2024-03-07_at_11.48.44_PM.png) - -Next, use the Javascript SDK to interact with your Xata database, by installing the Xata CLI globally using the following command: - -```bash -# Installs the CLI globally -npm install -g @xata.io/cli -``` - -Then, link the Xata project to your Express.js application by executing the following command in your terminal window: - -```bash -# Initialize your project locally with the Xata CLI -xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.us-east-1.xata.sh/db/cache-openai-responses -``` - -Use the following answers to the Xata CLI one-time setup question prompts to integrate Xata with Express.js: - -- `Yes` when prompted to add `.env` to `.gitignore`. -- `JavaScript require syntax` when prompted to select the language to generate code and types from your Xata database. -- `xata.js` when prompted to enter the output path for the generated code by the Xata CLI. -- `No` when prompted to generate the TypeScript declarations. - -![Setup Xata via CLI](/images/cache-openai-langchain/Screenshot_2024-03-07_at_11.52.35_PM.png) - -Great! Now, letā€™s move on to the creating an endpoint that caches ChatOpenAI Responses using Callbacks in LangChain. - -## Cache LangChain ChatOpenAI Responses using Callbacks - -Just like any other machine, a Large Language Model (LLM) response to a user query moves across various stages. It is required to start, it is then assigned a task to be done, and it completes the task. LangChain allows you to tap into these (asynchronous) stages while generating Chat responses with [Callbacks](https://js.langchain.com/docs/modules/callbacks/). Callbacks is a set of handlers that are used to intercept response creation by LLM at each stage. For example, `handleLLMStart` is called as soon as the chat model starts to process the user prompt. In this section, youā€™ll learn how to use [multiple handlers](https://js.langchain.com/docs/modules/callbacks/#multiple-handlers) that allow you specifically perform additional operations when a LLM starts, gets a new token and finishes responding. Letā€™s create a relevant schema for caching the responses during these stages of the LLM. - -### Creating the Schema - -Go to the **Schema** tab in your Xata database, and update the `responses` table to have two `text` fields, named `input` and `answer`. To do so, click on `+ Add a column` and select column type as `Text`. Finally, enter the column name and click `Create column`. - -![Make schema changes](/images/cache-openai-langchain/Screenshot_2024-03-08_at_1.08.52_AM.png) - -The `xata pull` command pulls down schema changes, generates a migration file along with client files that your application may require. Execute the following command to bring in changes from the schema updates we did above: - -```bash -xata pull main -``` - -This would update files in `xata.js` and `.xata/migrations` with the schema changes. - -### Creating a Cached ChatOpenAI Response Endpoint using Callbacks - -Letā€™s understand how to use the system of callbacks by LangChain by creating an endpoint `query-callback-method` which would respond to a user query using the `ChatOpenAI` LangChain model with callbacks and streaming enabled. - -First, install the dependencies required to start using LangChain in your Express.js by executing the following command in your terminal window: - -```bash -pnpm add @langchain/openai langchain -``` - -Then, add the following code snippet to the existing `index.js` file: - -```tsx -// File: index.js - -// ... - -const { ChatOpenAI } = require('@langchain/openai'); - -const { ConversationChain } = require('langchain/chains'); - -// ... - -app.post('/query-callback-method', async (req, res) => { - const { input } = req.body; - - // Set headers before piping the stream - res.setHeader('Transfer-Encoding', 'chunked'); - res.setHeader('Content-Type', 'text/plain; charset=utf-8'); - - const encoder = new TextEncoder(); - - const model = new ChatOpenAI({ - streaming: true, - callbacks: [ - { - async handleLLMStart() { - res.write(encoder.encode('[MISS] ')); - }, - handleLLMNewToken(token) { - res.write(encoder.encode(token)); - }, - async handleLLMEnd(output) { - // End the response - res.end(); - } - } - ] - }); - const chain = new ConversationChain({ llm: model }); - await chain.call({ input }); -}); - -// ... -``` - -The code above, creates an endpoint `/query-callback-method` which can be `POST`-ed to. Then, it destructures `input` from the request body. Then, it sets the `Content-Type` and `Transfer-Encoding` header which indicate creation of a streaming response from this endpoint to the client. Further, it creates a LLM via a `ChatOpenAI` instance with LangChain using multiple callbacks as follows: - -- **handleLLMStart**: Itā€™s called when the Chat Model begins working on a prompt -- **handleLLMNewToken**: Itā€™s called whenever the Chat Model in `streaming` mode generates a new token while processing a request -- **handleLLMEnd**: Itā€™s called when the Chat Model finishes processing a request to generate its final output - -Finally, it creates a single message conversation between the user and AI, and awaits the complete response by AI to the user query. - -To save repetitive calls to the OpenAI API that will mount onto your existing cost of running the chat responses, cache the user queries with their responses using Xata. To do so, perform the following additions in your `index.js` file: - -```tsx {5,9-10,26-37,44-48} -// File: index.js - -// ... - -+ const { getXataClient } = require("./xata"); - -// ... - -+ // Global Xata Client Instance -+ const xata = getXataClient(); - -app.post("/query-callback-method", async (req, res) => { - const { input } = req.body; - - // Set headers before piping the stream - res.setHeader("Transfer-Encoding", "chunked") - res.setHeader("Content-Type", "text/plain; charset=utf-8") - - const encoder = new TextEncoder(); - - const model = new ChatOpenAI({ - streaming: true, - callbacks: [ - { - async handleLLMStart() { -+ // Look for cached response in Xata -+ const cachedResponse = await xata.db.responses -+ .filter({ input }) -+ .select(["answer"]) -+ .getFirst(); - -+ // If cached response found, return as is -+ if (cachedResponse) { -+ res.write("[HIT] " + cachedResponse.answer); -+ res.end(); -+ return; -+ } - res.write(encoder.encode("[MISS] ")); - }, - handleLLMNewToken(token) { - res.write(encoder.encode(token)); - }, - async handleLLMEnd(output) { -+ // Once the response is sent, cache it in Xata -+ await xata.db.responses.create({ -+ input, -+ answer: output.generations[0][0].text, -+ }); - // End the response - res.end(); - }, - }, - ], - }); - const chain = new ConversationChain({ llm: model }); - await chain.call({ input }); -}); - -// ... -``` - -The code above, starts with importing the `getXataClient` method from the generated `xata.js` file. Then, it creates a global instance of the Xata client. Further, it updates the LangChain callbacks as follows: - -- handleLLMStart: As the LLM starts to respond to the user query, it first looks for a cached response in your Xata databaseā€™s `responses` table. This is implemented via the following: - -```tsx -// Look for cached response in Xata -const cachedResponse = await xata.db.responses.filter({ input }).select(['answer']).getFirst(); -``` - -The code above gets a single row matching the value of input of the user from the responses table in your Xata database. - -- handleLLMEnd: As the LLM is about to finish responding to the user query (if the response is not cached), itā€™ll cache the response in your Xata database using the following code: - -```tsx -// Once the response is sent, cache it in Xata -await xata.db.responses.create({ - input, - answer: output.generations[0][0].text -}); -``` - -Great! Now, letā€™s test if responses are being cached by executing the following command in your terminal: - -```bash -curl --location 'http://localhost:3005/query-callback-method' \ ---header 'Content-Type: application/json' \ ---data '{ - "input": "Who are you?" -}' -``` - -The output in your terminal window would start with a **\[MISS]** indicating that itā€™s not a cached response. - -![MISS response](/images/cache-openai-langchain/Screenshot_2024-03-08_at_2.41.13_AM.png) - -Now, execute the above command again to see the following output in your terminal window indicating that itā€™s a cached response returned from your Xata database. - -![HIT response](/images/cache-openai-langchain/Screenshot_2024-03-08_at_2.42.41_AM.png) - -Awesome! Youā€™re now successfully able to cache responses generated by OpenAI with Xata using LangChain callbacks. - -Now, letā€™s move to learning another way of caching OpenAI responses while using LangChainā€™s ChatOpenAI model. - -## Caching LangChain ChatOpenAI Responses using a Cache Layer - -In this section, youā€™ll learn how to use [LangChain Caching Layer](https://js.langchain.com/docs/modules/model_io/chat/caching) that allows you to override the implementation of how caches are built and searched for, for a given user query. Letā€™s create a relevant schema for caching the responses using the LangChain Caching Layer. - -### Creating the Schema - -Go to the **Schema** tab in your Xata database, and update the `invocations` table to have a `String` type field named `key` and a `Text` field named `answer`. To do so, click on `+ Add a column` and select column type as `String`. Finally, enter the column name as `key` and click `Create column`. Follow similar steps with the relevant type for creating the `answer` column. - -![Make schema changes](/images/cache-openai-langchain/Screenshot_2024-03-08_at_1.11.35_AM.png) - -As youā€™ve learned earlier, execute the following command to bring in changes from the schema updates we did above into your project locally: - -```bash -xata pull main -``` - -This would update files in `xata.js` and `.xata/migrations` with the schema changes. - -### Creating Cached ChatOpenAI Response Endpoint using the Cache Layer - -Letā€™s understand how to use the cache layer by LangChain by creating an endpoint `query-cache-method` which would respond to a user query using the `ChatOpenAI` LangChain model. - -First, install the dependencies required to start using a LangChain cache layer in your Express.js by executing the following command in your terminal window: - -```bash -pnpm add @langchain/core -``` - -Each cache layer in LangChain has two functions: - -- **lookup**: an asynchronous function that looks for responses to the user queries in a data source before generating a fresh response -- **update**: an asynchronous function that synchronizes the responses by OpenAI to an upstream after generating a fresh response - -Letā€™s start by creating a file named `langchain-xata-cache.js` that will override the default implementation of these two functions to set the upstream as Xata with the following code: - -```tsx -// File: langchain-xata-cache.js - -const { BaseCache, deserializeStoredGeneration, getCacheKey, serializeGeneration } = require('@langchain/core/caches'); - -class XataCache extends BaseCache { - // A Xata Client - xataClient; - - // Constructor for XataCache class - // Avails xataClient to class methods - constructor(props) { - super(); - this.xataClient = props.client; - } - - // A function to filter and return the - // cached response of model invocation - async makeValue(key) { - const tmp = await this.xataClient.db.invocations.filter({ key }).getFirst(); - if (tmp) return tmp.answer; - } - - /** - * Lookup LLM generations in cache by prompt and associated LLM key. - */ - async lookup(prompt, llmKey) { - let idx = 0; - let key = getCacheKey(prompt, llmKey, String(idx)); - let value = await this.makeValue(key); - const generations = []; - while (value) { - generations.push(deserializeStoredGeneration(JSON.parse(value))); - idx += 1; - key = getCacheKey(prompt, llmKey, String(idx)); - value = await this.makeValue(key); - } - return generations.length > 0 ? generations : null; - } - - /** - * Update the cache with the given generations. - */ - async update(prompt, llmKey, value) { - for (let i = 0; i < value.length; i += 1) { - const key = getCacheKey(prompt, llmKey, String(i)); - await this.xataClient.db.invocations.create({ - key, - answer: JSON.stringify(serializeGeneration(value[i])) - }); - } - } -} - -exports.XataCache = XataCache; -``` - -The code above creates a `XataCache` class that extends LangChainā€™s `BaseCache` class to help override the following two cache functions. Then, it creates a constructor of the class that accepts a Xata instance while a new object is being created. Then, it overrides the `lookup` function to [filter for responses](https://xata.io/docs/sdk/filtering) in your Xata databaseā€™s `invocations` table using the `makeValue` function. Finally, it overrides the `update` function to cache responses by [inserting a record](https://xata.io/docs/sdk/insert) in your Xata databaseā€™s `invocations` table. - -To use this cache class, add the following code snippet to the existing `index.js` file to create an endpoint `/query-cache-method`: - -```tsx -// File: index.js - -// ... - -// XataCache class built on top of LangChain's BaseCache -const { XataCache } = require('./langchain-xata-cache.js'); - -const { ChatOpenAI } = require('@langchain/openai'); - -const { getXataClient } = require('./xata'); - -// Global Xata Client Instance -const xata = getXataClient(); - -// ... - -app.post('/query-cache-method', async (req, res) => { - const { input } = req.body; - - // Use LangChain Xata Cache Adapter - const model = new ChatOpenAI({ - cache: new XataCache({ client: xata }) - }); - - console.time(); - - // Perform a cached-response first AI model invocation - const response = await model.invoke(input); - - console.timeEnd(); - - res.write(response.content); - res.end(); -}); - -// ... -``` - -The code above, starts with importing the `XataCache` class that we created earlier. Then, it creates an endpoint `/query-cache-method` which can be `POST`-ed to. Then, it destructures `input` from the request body. Then, it creates a LLM via a `ChatOpenAI` instance with LangChain. The `cache` property of the ChatOpenAI model is set to a new instance of `XataCache` using the existing global Xata client. Then, it invokes the model to respond to the user query. Finally, the content of the response by OpenAI is returned to the user. - -Great! Now, letā€™s test if responses are being cached by executing the following command in your terminal: - -```bash -for i in {1..9}; do - curl --location 'http://localhost:3005/query-cache-method' \ - --header 'Content-Type: application/json' \ - --data '{ - "input": "Who are you?" - }' - echo "Execution $i completed." -done -``` - -The output in your terminal window would indicate the time (approximately) it took to respond to the user queries. An immediate decrease in the response times after the very first one indicates cached responses being returned for the user queries. - -![Multiple Queries Time Performance](/images/cache-openai-langchain/Screenshot_2024-03-08_at_3.12.34_AM.png) - -Awesome! Youā€™re now successfully able to cache responses generated by OpenAI with Xata using the LangChain Caching Layer. - -## Final Thoughts - -In this guide, you learned different ways of caching OpenAI API responses using LangChain callbacks and the caching layer. Using Xata, youā€™re able to interact with a Postgres serverless database to build a set of cache OpenAI API responses, and utilize Xataā€™s search capabilities to respond to user queries faster using those cache responses. - -For more detailed insights, explore the references cited in this post. - -| Resource | Link | -| ------------------------------- | ------------------------------------------------------------- | -| GitHub Repo | https://github.com/rishi-raj-jain/xata-cache-openai-langchain | -| Caching - LangChain Chat Models | https://js.langchain.com/docs/modules/model\_io/chat/caching | -| Callbacks - LangChain Modules | https://js.langchain.com/docs/modules/callbacks/ | - -## Whatā€™s next? - -We'd love to hear from you if you have any feedback on this guide, would like to know more about Xata, or if you'd like to contribute a community blog or tutorial. Reach out to us on [Discord](https://xata.io/discord) or join us on [X | Twitter](https://twitter.com/xata). Happy building šŸ¦‹ diff --git a/chatgpt-on-your-data-2.mdx b/chatgpt-on-your-data-2.mdx deleted file mode 100644 index 266fd60c..00000000 --- a/chatgpt-on-your-data-2.mdx +++ /dev/null @@ -1,201 +0,0 @@ ---- -title: 'Building a chatbot with OpenAI, Vercel AI and Xata' -description: 'Ask your data questions and get intuitive, efficient answers with OpenAI, Vercel AI and Xata' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/xata_vercel_chatgpt.jpeg - alt: Xata, Vercel and ChatGPT logos -author: Alexis Rico -authorEmail: alexis@xata.io -date: 09-06-2023 -tags: ['ai'] -published: true -slug: openai-vercel-xata-tutorial -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/xata_vercel_chatgpt.jpeg ---- - -In today's data-driven world, efficient interaction with databases is a crucial aspect of many applications. But what if we could go beyond conventional search methods and enable a natural language conversation with our databases? - -At Xata, we aim to provide developers with the tools to build powerful applications that can interact with data in a natural way. Built-in with our core APIs and SDKs, we offer a powerful [ask endpoint](https://xata.io/docs/sdk/ask) that allows you to ask questions about your data and get the answers that matter most. - -However, how would you integrate Xata with an existing application built with OpenAI? In this tutorial, we'll show you how to integrate OpenAI's [function calling](https://platform.openai.com/docs/guides/gpt/function-calling) feature with Xata's TypeScript SDK to create a chatbot that can use search to answer questions about your data. - -In a [previous post](https://xata.io/blog/chatgpt-on-your-data), we introduced the concept of using our built-in [ask endpoint](https://xata.io/docs/sdk/ask) to simplify the process of querying your data. - -## Prerequisites - -Before we begin, make sure you have the following prerequisites: - -- Existing project configured with the [Xata CLI](https://xata.io/docs/getting-started/installation) -- [Xata API key](https://app.xata.io/settings) -- [OpenAI API key](https://platform.openai.com/account/api-keys) - -In your preferred serverless environment, make sure you install the [OpenAI API Library](https://github.com/openai/openai-node) and [Vercel AI library](https://github.com/vercel-labs/ai) to get started. - -After ensuring your prerequisites are met, you can integrate Xata with your existing OpenAI application in three steps: Define a search function for AI, ask questions about your data, and run completions while streaming the results. - -## Step 1 - Defining a search function - -First, we'll define a function that allows us to search our database using OpenAI's [function calling](https://platform.openai.com/docs/guides/gpt/function-calling) feature. - -```ts -const functions: CompletionCreateParams.Function[] = [ - { - name: 'full_text_search', - description: 'Full text search on a branch', - parameters: { - type: 'object', - properties: { - query: { - type: 'string', - description: 'The search query' - } - }, - required: ['query'] - } - } -]; -``` - -If we want to allow OpenAI to fine-tune the search results, we can add more options to the `parameters` object. For example, we can add a `fuzziness` parameter to allow for fuzzy search. - -```ts {12-15} -const functions: CompletionCreateParams.Function[] = [ - { - name: 'full_text_search', - description: 'Full text search on a branch', - parameters: { - type: 'object', - properties: { - query: { - type: 'string', - description: 'The search query' - }, - fuzziness: { - type: 'number', - description: 'Maximum levenshtein distance for fuzzy search, minimum 0, maximum 2' - } - }, - required: ['query'] - } - } -]; -``` - -## Step 2 - Ask a question about your data - -Now that we have our search function defined, we can use the [OpenAI](https://www.npmjs.com/package/openai) library to ask a question about our data. - -```ts -const openai = new OpenAI({ - // Make sure to properly load and set your OpenAI API key here - apiKey: process.env.OPENAI_API_KEY -}); - -const model = 'gpt-3.5-turbo'; - -const response = await openai.chat.completions.create({ - model, - messages: [ - { - role: 'user', - content: question - } - ], - functions -}); -``` - -With the functions defined, the AI will be able to call the `full_text_search` function and pass the `query` parameter with the parts of the question that are relevant to the search. - -To enhance the results, we can include additional information in the `messages` array as system messages. For instance, we can provide instructions to the AI or offer hints related to our database. - -```ts {1,11-23} -const { schema } = await api.branches.getBranchDetails({ workspace, region, database, branch }); - -const response = await openai.chat.completions.create({ - model, - stream: true, - messages: [ - { - role: 'user', - content: question - }, - { - role: 'system', - content: ` - Workspace: ${workspace} - Region: ${region} - Database: ${database} - Branch: ${branch} - Schema: ${JSON.stringify(schema)} - - Reply to the user about the data in the database, do not reply about other topics. - Only use the functions you have been provided with, and use them in the way they are documented. - ` - } - ], - functions -}); -``` - -OpenAI provides a variety of models, which you can find listed [here](https://platform.openai.com/docs/models/overview). If you require a different model that aligns more closely with your specific use case, you can easily switch the `model` parameter. - -## Step 3 - Running the completion and streaming the results - -Finally, we can run the completion and stream the results to the client. - -```ts -const stream = OpenAIStream(response, { - experimental_onFunctionCall: async ({ name, arguments: args }, createFunctionCallMessages) => { - switch (name) { - case 'full_text_search': { - const response = await api.searchAndFilter.searchBranch({ - workspace, - region, - database, - branch, - query: args.query as string, - fuzziness: args.fuzziness as number - }); - - const newMessages = createFunctionCallMessages(response); - - return openai.chat.completions.create({ - messages: [...messages, ...newMessages], - stream: true, - model, - functions - }); - } - default: - throw new Error('Unknown OpenAI function call name'); - } - } -}); - -return new StreamingTextResponse(stream); -``` - -We have chosen to stream the results to the client, but you can also wait for the completion to finish and return the results as a single response. - -### Bonus - Building an interactive chatbot UI - -With React and the [`useChat` hook](https://sdk.vercel.ai/docs/api-reference/use-chat) you can easily create a chatbot that can answer questions about your data. - -```tsx -const { messages, append, reload, stop, isLoading } = useChat({ - // The route to the endpoint we have just created - api: '/api/chat', - initialMessages -}); -``` - -### Conclusion - -Congratulations! You've just built a powerful system that combines the capabilities of OpenAI, Vercel AI and Xata's database API. Users can now engage in natural language conversations with their databases, and OpenAI will utilize the provided functions to perform searches and retrieve relevant information. - -By following this tutorial, you've learned how to integrate multiple APIs, handle requests and create interactive responses. This foundation can be extended to create even more sophisticated systems that enable seamless human-machine interactions with data. - -If you want to learn more about Xata's database API, check out our [documentation](https://xata.io/docs) and come say hi on our [Discord](https://xata.io/discord) if you have any questions. - -Happy coding! šŸ¦‹ diff --git a/chatgpt-on-your-data.mdx b/chatgpt-on-your-data.mdx deleted file mode 100644 index 01ceafdf..00000000 --- a/chatgpt-on-your-data.mdx +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: 'Xata & OpenAI: ChatGPT for your data' -description: 'Use Xata to ask your data questions and get the answers that matter most.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/chatgpt-on-your-data.png - alt: Xata and ChatGPT logos -author: Daniel Everts -date: 03-02-2023 -published: true -tags: ['ai', 'fpGenerativeAi'] -slug: chatgpt-on-your-data ---- - -Xataā€™s here to take care of the hard stuff surrounding data: high availability, scaling, access control, synchronization, and a bunch more. Our data platform is built to handle these problems, so you can focus on building whatā€™s unique to your product. As developers ourselves, building a platform is fun. We use Xata in our own work as much as possible, and out of it comes a tonne of new ideas. These ideas, turned into features, help you move faster. - -It's been hard to miss the thunderstorm generated by OpenAI. For the last few months, itā€™s been everywhere; simple answers to tricky questions, hip-hop songs about Kubernetes, or standup comedy about functional programming. From all the uses, one thing felt clear: this was a firm shove into the future. - -With their APIs, we now have tools to help you analyze, reshape, and reframe text. Tools to help inspire new ideas, distill information down into ELI5-chunks, or generate photo-realistic pictures of cats in space. It provides an API that deals with a complicated world. One where you canā€™t easily prepare for all reasonable inputs. - -This isnā€™t a teary-eyed post about OpenAI though. Weā€™re wanting to show you something weā€™ve had brewing over the last few weeks. Letā€™s begin with a screenshot taken a couple of days ago, from ChatGPT. - - - -For those familiar with Xata, you'll see that this not correct. ChatGPT has confidently given us a wrong answer. We canā€™t blame it though, the model has a documented cutoff date for its data in 2021 and unfortunately for us, Xataā€™s documentation was published after this date. - -Letā€™s take a look at once more image. Iā€™ll ask the same question again. - - - -Weā€™ve been working on integrating OpenAI into our data platform. Xata now lets you ask your data questions. It relies fully on the data you have inside Xata. No waiting for OpenAI, Google, or any other company to index your changes. Itā€™s your data, and youā€™re fully in control of it. You choose when to add information, remove information, modify information. Xata takes these changes and makes them available nearly instantly. It requires no work from you, and comes with options for both free and pro users. - -The idea came out of a [demo day](https://twitter.com/xata/status/1626631119306096658?s=61&t=kJ96Ir__sVvypRcu-eaOPw). We run a session every few weeks where the team gathers to show off ideas, proof-of-concepts, or features weā€™ve just completed. During the demo for the integration (in those days it was called DanGPT), I had my screen shared. Demos are bad at the best of times, and here, I wasnā€™t totally sure what would be returned. I wrote my question, and nervously pressed a green ā€œAskā€ button that I had only finished that morning. The result that came out was almost perfect. It needed tuning, but seeing it take a natural-sounding question and coming back with Xata-specific details felt like magic. - -Building this feature has been the source of a buffet of new ideas for us. Hereā€™s a few things weā€™re thinking about. - -## Improve your documentation search - -We use Xata to store our documentation. This is stored as markdown in `text` columns. The Xata Ask API queries the data and returns a response that matches the context of the question and uses the content we have stored. This helps users find information quicker, make use of it faster, and to be able to ask questions in a way that feels natural to them, rather than needing to try and adapt their question to what they think our docs wants. - -This is coming to our documentation, look out for the ā€œAsk our chat botā€ buttons on the documentation site, and in the Xata app. - -## Help your team get the information they need faster - -If you have a lot of data, finding the right document can be tough. Finding the right text in that document can also be tough. A question might be phrased in a way that is uncommon, or an answer might require context from several documents. - -Xata understands the intent of the question, and has to one column of one table all the way up to all columns in all tables. This means that it can take a wide range of questions, and stitch together a coherent answer. - -## Augment customer support - -Have a FAQ or receive common questions? Use Xata to help answer common questions before being escalated to an agent. The responses will adapt to the questions, and you can write documentation in natural language. Xataā€™s search service will make the data you change available almost immediately. - -Whatā€™s got us excited about releasing our integration with OpenAI is that we can offer it on top of our existing platform. By using one of our SDKs or the REST API, youā€™re able to leverage this technology without any major changes to your workflow. - -Hereā€™s the API call our docs make to retrieve data from our docs table. - -```jsonc -// POST https://xata-uq2d57.eu-west-1.xata.sh/db/docs:main/tables/search/ask -// Accept: text/event-stream -{ - "question": "How do a retrieve a single record?", - "rules": [ - "Do not answer questions about pricing or the free tier. Respond that Xata has several options available, please check https://xata.io/pricing for more information.", - "When you give an example, this example must exist exactly in the context given.", - "Only answer questions that are relating to the defined context or are general technical questions. If asked about a question outside of the context, you can respond with \"It doesn't look like I have enough information to answer that. Check the documentation or contact support.\"", - "Your name is DanGPT" - ], - "searchType": "keyword", - "search": { - "fuzziness": 2, - "prefix": "phrase", - "target": ["slug", { "column": "title", "weight": 4 }, "content", "section", { "column": "keywords", "weight": 4 }], - "boosters": [ - { - "valueBooster": { - "column": "section", - "value": "guide", - "factor": 18 - } - } - ] - } -} -``` - -[Sign up](https://app.xata.io) to add ChatGPT to your data today and get started with this new [sample app](https://github.com/xataio/examples/tree/main/apps/sample-chatgpt). Come say hi on our [Discord](https://xata.io/discord) if you have any questions. - -Weā€™re excited to see what you build šŸ¦‹ diff --git a/data-modeling-in-typescript.mdx b/data-modeling-in-typescript.mdx deleted file mode 100644 index adb2608c..00000000 --- a/data-modeling-in-typescript.mdx +++ /dev/null @@ -1,270 +0,0 @@ ---- -title: 'The importance of data modeling in TypeScript' -description: 'Take a deep dive into TypeScript to explore the importance of data modeling.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/the-importance-of-data-modeling-in-typescript.png - alt: 'The importance of data modeling in TypeScript' -author: Fabien Bernard -date: 03-28-2023 -tags: ['typescript'] -published: true -slug: the-importance-of-data-modeling-in-typescript ---- - -## Introduction - -Hey folks! Today we will deep dive into TypeScript to demonstrate the importance of data modeling. - -Quite often, I found myself having to come up with very fancy solutions for solving what I thought was a simple problem. Even if itā€™s good practice (and sometimes quite fun), most of the time, this is a sign that the data modeling can be improved! - -Welcome to my journey of implementing a ā€œstrategy patternā€ in TypeScript! - -Whatā€™s this?! Itā€™s a ā€œcomputer science termā€ to describe that you need to choose a strategy on runtime. Still donā€™t get it? Letā€™s jump into an example to see what we are trying to solve here. - - - -We need to build this `Settings` component, this component will take a `question` as input, andā€¦ this question can be 1 of 18 question types. - -To make this a bit more challenging, every question type has its own set of settings. - -This is where the ā€œstrategy patternā€ comes in. Depending on the question type, at runtime, I will choose a different strategy/component to display. So if I pass a `ContactInfoQuestion` I will render `ContactInfoSettings`, if I pass a `NumberQuestion` I want `NumberSettings`, and so on. - -Also, to help me implement all those `{questionType}Settings` components, I want to have the most precise type I can. For example, `NumberSettings` will take a `NumberQuestion` as props, so I only have the relevant information about in the context of `NumberSettings`. - -## Letā€™s Summarize - -In code, this is what I want: - -```tsx -function Settings(props: { question: Question }) { - switch (question.type) { - case 'contact': - return ; - case 'number': - return ; - } -} - -function SettingContact(props: { question: ContactQuestion }) { - /* I just have the context of the question of type === `contact` */ -} -function SettingNumber(props: { question: NumberQuestion }) { - /* I just have the context of the question of type === `number` */ -} -``` - -Easy, isnā€™t it? It depends! Here is where we must stop and think about our data modeling. - -Indeed, the shape of `Question` will define everything here, between a hardcore TypeScript problem, with type guards, `infer`, ternaries and so on, or something smooth that works out of the box. - -## Without Discriminator - -Letā€™s start with the naive approach without any discriminator. Donā€™t you know what a discriminator is? Perfect, we are not using them! (donā€™t worry, I will come back to it šŸ˜…) - -Here is our first iteration of the type `Question`: - -```bash -type Question = - | { - title: string; - required: boolean; - min?: number; - max?: number; - } - | { - title: string; - required: boolean; - format: string; - separator: string; - }; -``` - -This is how to start a painful journey with TypeScript, just because you donā€™t have any discriminator! Indeed, how do I know what is what? And if I donā€™t know, how does TypeScript know? I have no explicit value to know that our first option is `QuestionNumber` and our second is a `QuestionDate` - -But, this is solvable with TypeScript, using type guards like this: - -```tsx -function isQuestionNumber(question: Question): question is QuestionNumber { - return 'min' in question && 'max' in question; -} -``` - -My advice is if you have this from your backend team, talk to them and fix the problem at the source! Really, you donā€™t want to do this to yourself! - -Now letā€™s add this famous ā€œdiscriminatorā€, but for some reason, in a `metadata` object (you know, to ā€œkeep this cleanā€, and yes, I made this mistake before šŸ˜“) - -```tsx -type Question = - | { - title: string; - required: boolean; - min?: number; - max?: number; - metadata: { - type: 'number'; - }; - } - | { - title: string; - required: boolean; - format: string; - separator: string; - metadata: { - type: 'date'; - }; - }; -``` - -Please note that we are using string literals here (`ā€dateā€` and `"number"`) and not `string`, this is really important! Thanks to this added string literal, we can ā€œeasilyā€ narrow our type, we can discriminate! - -Letā€™s try! - - - -Yesā€¦ I knowā€¦ Disappointingā€¦ if you have this, out-of-box, TypeScript will still not help you, so there are two choices: - -- Long party of writing type-guards ā†’ Boring and tedious -- Add a little `as QuestionDate` and we are good to go ā†’ Error-prone and we are bypassing Typescript - -This is usually when you start saying that TypeScript is stupid, and yes, in this case, you are right! And maybe, in future updates of TypeScript, this scenario will work! - -## Letā€™s Fix This - -The fix is actually quite easy, we just need to move the `type` (our discriminator) to the top level: - -```tsx -type Question = - | { - type: 'number'; - title: string; - required: boolean; - min?: number; - max?: number; - } - | { - type: 'date'; - title: string; - required: boolean; - format: string; - separator: string; - }; -``` - - - -And voilĆ ! Without any TypeScript guru skills required, you have exactly what you want! - -Remember: If you want TypeScript to discriminate, the discriminator needs to be top-level! - -Also, keep in mind that when you have a tough problem to solve with TypeScript, challenge the data modeling! - -## Component Part - -Our switch statement is working, but what about our component prop? - -To keep it simple, we can split our `Question` into multiple types - -```bash -type NumberQuestion = { - type: "number"; - title: string; - required: boolean; - min?: number; - max?: number; -}; - -type DateQuestion = { - type: "date"; - title: string; - required: boolean; - format: string; - separator: string; -}; - -type Question = NumberQuestion | DateQuestion; -``` - -Doing so, we can use `NumberQuestion` type as props in my component, which is simple and easy. - -But wait! we can be smarter! So we donā€™t have to extract every union subtype (because, yes, we always realize this after writing everything in line šŸ™ƒ). Letā€™s use a type helper šŸŖ„ - -```tsx -export type QuestionType = Question['type']; -export type QuestionOfType = U extends { - type: T; -} - ? U - : never; -``` - -First, we extract the `type` from our union, `QuestionType` will be `"number"|"date"` in our reduce example. So every time we add a new question type in my union, this will follow. - -Second, and this is where this is getting interesting, `QuestionOfType<>`! - -We are using this `QuestionType` to constrain our generic. This gives us autocompletion later when we will use the helper and give guidance. `T` is our input, like a function parameter. - -We also need a `U` , but we donā€™t want this as an entry, this is why we are giving the `Question` type as default. Also, you might ask why do we need a generic at all? This is to allow TypeScript to narrow the resulting type. Indeed, if I remove this `U` and write `Question extends {type: T} ? Question : never` This will always result in `never` since we are evaluating `{type: T}` on the entire `Question` instead of a shape. - -Lastly, the logic: - -If the type `U` extends `{ type: T }`, returns `U`; otherwise, returns `never` . This is narrowing our union type to the subtype that extends `{ type: T }` , basically discriminating on `type` ! - -Letā€™s try to use this new helper in our component - -```tsx -export default function SettingsDate(props: { question: QuestionOfType<'date'> }) {} -``` - -Both approaches are totally valid. But the second one brings us something interesting, and we can use the `question.type` as generic. - -In my context, my `Settings` component needs two more things: `formId` and `questionId` and this is where we can utilize this generic to create a standard `SettingsProps` - -```tsx -export type SettingsProps = { - question: QuestionOfType; - questionId: string; - formId: string; -}; -``` - -I now have a simple and robust way to share props across all our components - -```tsx -export default function SettingsDate({ questionId, formId, question }: SettingsProps<'date'>) {} -``` - - - -How beautiful is this? šŸ˜ No guessing, no type assertion, and this without writing (and maintaining) 18 type guards! - -I hope you did learn something today! If not, thanks for following my journey šŸ˜€ All the examples in this article are taken from XataForm, my in-progress project that should soon be open-source. - -If you have any questions, suggestions, remarks, or just want to say hi, Iā€™m always around in our Discord channel! (@fabien0102) diff --git a/datetime-picker.mdx b/datetime-picker.mdx deleted file mode 100644 index 53cbc192..00000000 --- a/datetime-picker.mdx +++ /dev/null @@ -1,300 +0,0 @@ ---- -title: 'Building a datetime picker for a database' -description: 'Learn how we built a next-generation datetime picker for a next-generation database.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/date-time-picker-for-databases.png - alt: Xata -author: Richard Gill -date: 04-05-2023 -tags: ['engineering', 'product'] -published: true -slug: date-time-picker-for-databases ---- - -Xata, our modern database, supports datetime columns. - -When we launched last year we decided to just use the native browser date picker. This date picker has a number of issues, and we just werenā€™t happy with it. - -## Datetimes in Xata - -In Xata, datetimes returned from our API look like this: `2000-01-01T01:02:03.123Z` - -Lets break down all the parts: - -```txt - month hour second - āˆØāˆØ āˆØāˆØ āˆØāˆØ -"2000-01-01T01:02:03.123Z" - āˆ§āˆ§āˆ§āˆ§ āˆ§āˆ§ āˆ§āˆ§ āˆ§āˆ§āˆ§ - year day minute millisecond -``` - -All of Xataā€™s datetimes are saved and returned with the UTC timezone. - -## Who uses our date picker? - -Xata has two main personas: - -- Developers -- Low-code users - -### Developers - -At itā€™s core, Xata is a database aimed at developers. - -Developers are power users: Theyā€™re comfortable with not leaving their keyboard, they understand datetime strings, and, most of all, they love and respect the UTC timezone. - -### Low-code users - -Entrepreneurs, hackers, customer success, you name it. - -They use Xataā€™s Web UI to quickly view their data, add records and make edits. - -They might be more likely to prefer using a date picker widget with the mouse or paste a value in from google sheets or airtable. - -## Use cases - -There are a few scenarios when a user selects a datetime inside our WebUI: - -### User has a specific date theyā€™d like to set - -This might be with a precision of: - -- Day: e.g. `2000-01-01` -- Time: e.g. `2000-01-01T01:02:03` -- Millisecond: e.g. `2000-01-01T01:02:03.123` - -### Pasting - -Theyā€™re pasting a datetime from another source - -- e.g. `25/01/2000` pasted from Excel -- e.g. `2nd Mar 2015 23:22:00` pasted from another system. - -### Any date is fine - -- If theyā€™re testing something or adding dummy data - -### User has a relative or human time in mind - -- `Next wednesday at 3pm` - -## Our new date time picker - -Here is the new date timer picker Dave Snider and I built. In glorious UTC. - - - -## Opening the date picker - -The datetimes in Xata are shown inside an ``. As soon as the input is focused, we pop up the picker. - - - -We close the the picker when focus moves away from the picker and input. - -## Setting the date - -Selecting a date in our datetime picker is easy and straightforward. Users can jump back months or select the month and year from the dropdowns. - - - -Notice that when a user selects the date, the value of the input changes to the new date. - -## Setting the time - -To set the time, we have built our own component with one input each for hour, minute, second, and millisecond. - -As the user completes each input, focus automatically shifts to the next one. - -Here the user is typing: `1 2 3 4 5 6 7 8 9` - - - -When a user focuses on an input, all of the text in the input is selected. This means that as soon as they start typing, they will overwrite the existing value in the input. - -These inputs have strict validation, which makes it impossible to enter an invalid time. Leaving an input empty sets it to `00`. - - - -Users can navigate between inputs using the tab key as expected. - - - -As the user enters their time, the value of the input with the datetime updates as well. - -## Typing in the input - -If users prefer, they can use the main input to edit their datetime directly. - - - -As they type text the datetime picker components update to show the correct values - itā€™s two-way connected. - -Typing directly into the input is good for a couple of reasons: - -- Itā€™s quick, users can use their keyboard to make a quick edit and press enter or tab to save it -- It allows us to support more formatsā€¦ - -### Supporting more formats - -When entering a datetime, it's possible itā€™s come from another data source. That data source likely uses a different format to Xata. - -Our date picker supports nearly all common date formats. Here are some examples you can paste or type in: - -- `15/5/2000` -- `15.05.2000 14:30` -- `may 15th 2000` - -This is great if you are copying data from Google Sheets or Excel and pasting it into Xata. - -We also support human readable formats: - -- `tomorrow at 4pm` -- `next wednesday at 15:00` -- `now` - - - -When you type the custom format youā€™ll see the date and time update automatically. When the date picker loses focus the input displays the UTC time again: `2000-05-15T14:00:00.000Z` - -The value the user sees inside the main input is exactly what gets saved inside Xata. - -It could be hard for users to discover this feature. To help with this weā€™ve added a couple of buttons at the top of the picker to help the user quickly try it out with a couple of examples. - -## How we built it - -### The Date picker - -There are several off-the-shelf date pickers we could have used, but we chose to use the excellent [dayzed library](https://github.com/deseretdigital/dayzed), which provides helpful React hooks for building a calendar widget. At Xata, we use [Chakra](https://chakra-ui.com/) as our component library, so it was quick and easy to build the rest of the calendar using it. - -### The Time picker - -We couldn't find an off-the-shelf time picker that worked the way we wanted. We wanted users to be able to quickly edit just one component of the time, which is not a common capability among many time pickers. Additionally, our requirement for milliseconds is somewhat niche. - -The time picker uses regular expressions to evaluate whether a partially or fully entered value is valid. - -Here is our code to validate hours: - -```tsx -const HOUR_REGEX = /^([0-1][0-9]|2[0-3])$/; -const ANY_DIGITS_REGEX = /^(\d)+$/; - -export const isValidHour = (hours: string) => { - return HOUR_REGEX.test(hours); -}; - -export const isValidPartialHour = (hours: string) => { - if (hours === '') { - return true; - } - if (hours.length === 1) { - return ANY_DIGITS_REGEX.test(hours); - } - return isValidHour(hours); -}; -``` - -Here is the code that decides whether to update the value and moves the focus to the next input if the cursor is at the end of the input: - -```tsx - { - const newValue = e.target.value; - const cursorIndex = e.target.selectionStart ?? 0; - if (isValidPartialHour(newValue)) { - onChange(newValue); - if (newValue.length === 2 && cursorIndex === 2) { - jumpToNextInput(); - } - } - }} -/> -``` - -### Parsing any datetime string - -We use [Chrono](https://github.com/wanasit/chrono), which makes it easy to parse almost any datetime. - -```tsx -import * as chrono from 'chrono-node'; - -chrono.parseDate('tomorrow at 3pm'); -// => Fri March 30 2023 15:00:00 -chrono.parseDate('Fri March 30 2023 15:00:00'); -// => Fri March 30 2023 15:00:00 -``` - -One modification we made to the default Chrono behaviour is to throw away any time information that is implied and only keep known values: - -```tsx -chrono.parse('tomorrow')[0].start; -// => { -// date: '2023-03-24T01:02:03.123Z, -// knownValues: {day: 24, month: 3: year 2023}, -// impliedValues: {hour:1 , minute:2, second: 3, millisecond: 123} -// } -``` - -In this example, the user said "tomorrow" without specifying any time information. Therefore, we discard the implied values and only use the known values, producing: March 24th, 2023 at 00:00:00.000 UTC. - -### Timezones (are hard!) - -One difficulty throughout this project is that timezones are hard. There is no timezone silver bullet, but hereā€™s some advice: - -- Unit test! - - Run your unit tests manually with `TZ="Europe/Berlin" pnpm test` to verify they still pass. -- Manually test by setting your system clock to a different timezone. -- Test daylight savings by setting dates in your picker that are after the clocks change. - -## Whatā€™s next? - -After shipping our v2 DateTime picker, our next goal is to allow users to save their preferred DateTime format and timezone. - -For example: - -Users can change their preferences in our web UI to display all datetimes in the format `29.05.2023 3pm`, presented in their local timezone instead of UTC. - -To do this we need to figure out how to allow users edit datetimes in the picker which currently assumes everything is in UTC or restrict users to editing datetimes in UTC. diff --git a/feature-flags-tutorial.mdx b/feature-flags-tutorial.mdx deleted file mode 100644 index a8372d39..00000000 --- a/feature-flags-tutorial.mdx +++ /dev/null @@ -1,256 +0,0 @@ ---- -title: 'Add feature flags in Astro apps using Xata and Vercel' -description: 'Follow this tutorial to learn how to implement feature flags in your Astro apps with Xata and Vercel.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/feature-flags-astro.png - alt: Create flags with Xata, Vercel, and Astro -author: Rishi Raj Jain -tags: ['guest-post', 'engineering', 'tutorial'] -date: 12-12-2023 -published: true -slug: feature-flags-tutorial -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/feature-flags-astro.png ---- - -In this tutorial, we're going to show you how to incorporate feature flags into your Astro app using Xata and Vercel. Imagine you're working on a feature, and you want to use email addresses to define the controlled rollout of the feature. We'll guide you through setting up these feature flags in Xata, allowing for a controlled rollout of new features. This enables you to gather feedback from a select group of users before launching it more broadly. So, let's get started on enhancing your app's functionality. - -### What are feature flags? - -In this tutorial, we'll show you how to use feature flags in your Astro app with Xata and Vercel. Imagine you're working on a feature and want to control its release using email addresses. We'll guide you through setting up these feature flags in Xata, allowing for a controlled rollout of new features. This way, you can gather feedback from a select group of users before a broader release. Let's get started: - -### Before you begin - -#### Prerequisites - -- You will need a [Xata account](https://app.xata.io/signin) -- [Node.js 18](https://nodejs.org/en/blog/announcements/v18-release-announce) or later - -#### Tech stack - -| Tool | Description | -| ---------------------------------------- | ---------------------------------------------------------------------------------------------------- | -| [Xata](https://xata.io) | A serverless database platform for data management. | -| [Astro](https://astro.build) | A framework that combines frontend and serverless backend capabilities for building modern web apps. | -| [Tailwind CSS](https://tailwindcss.com/) | A CSS framework for rapid and responsive styling. | -| [Vercel](https://vercel.com/) | A cloud platform for deploying and scaling web applications. | - -### Configuring Xata - -Start by [creating a Xata account](https://app.xata.io/signin), then log in to create a database. - -![](/images/feature-flags/signin_window.png) - -After creating your database, you'll need to make a table for your feature flag attributes. - -![](/images/feature-flags/create_database.png) - -Great! Now that you've completed that step, you should see the table in its default state, as shown below šŸ‘‡šŸ» - -![](/images/feature-flags/table_created.png) - -Let's move on to adding the feature flag with itā€™s relevant properties in the table youā€™ve created. - -### Creating feature flags in Xata - -In this particular example, youā€™ll create a feature flag that allows us to process email domains, such as `@someone.com`, `@gmail.com`, etc. and define a percentage number so that the feature flag can be rolled back whenever required. - -First, add the `percentage` attribute. Click **+** and select **Integer**. - -![](/images/feature-flags/create_feature_flags_1.png) - -Set the column name to `percentage` and avoid any null values. Select the **Not null** option and set the default value to 51. - -![](/images/feature-flags/create_features_flag_2.png) - -Great! Now, letā€™s add the `contains` attribute. By setting up a `contains` attribute, you can target specific conditions or criteria within your app. For instance, if the attribute is set to `@someone.com`, you can tailor the app's behavior for users with email addresses containing this domain. To add the attribute, click **+** and select **String** from the dropdown. - -![](/images/feature-flags/create_features_flag_3.png) - -Set the column name to `contains` and again avoid any null values. Select the `Not null` option. Choosing the string data type allows you to specify text-based conditions. Marking the field as `Not null` ensures that every record in the database has a value for this attribute, maintaining the consistency and reliability of your data. Set a default value of, for example, `@someone.com`. - -![](/images/feature-flags/create_features_flag_4.png) - -Awesome! Weā€™re now done with configuring the feature flag table and now can use it in your frontend to customize the onboarding flow. - -### Setting up the project - -Fork the repository by navigating to the original GitHub repository at https://github.com/rishi-raj-jain/adding-feature-flags-to-your-astro-app-with-xata-and-vercel. -Click **Fork** at the top right corner of the page. This creates a copy of the repository in your GitHub account. - -Once the repository is forked, clone it to your local machine. Replace `your-username` with your GitHub username in the following command: - -```sh -git clone https://github.com/rishi-raj-jain/adding-feature-flags-to-your-astro-app-with-xata-and-vercel -cd adding-feature-flags-to-your-astro-app-with-xata-and-vercel -npm install -``` - -## Configure Xata with Astro - -To use Xata with Astro, install the Xata CLI globally: - -```sh -npm install @xata.io/cli -g -``` - -Authorize the Xata CLI to associate with the logged in account: - -```sh -xata auth login -``` - -![](/images/feature-flags/add_API_key.png) - -Great! Initialize the project locally with the Xata CLI command with something like what is presented below: - -```sh -xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.us-east-1.xata.sh/db/feature-flags-with-astro -``` - -Lastly, answer some quick prompts to make the integration with Astro seamless for you. - -![](/images/feature-flags/features_terminal.png) - -### Configure the Astro frontend - -In this section, you'll learn how to customize an onboarding flow using feature flags set in Xata. You'll implement form actions in Astro routes, fetch feature flag from Xata, and define your own onboarding logic. - -#### Implement form actions in Astro - -With Astro, you can also allow [transitions on form submissions](https://docs.astro.build/en/guides/view-transitions/#transitions-with-forms). - -Enable form actions with view transitions in `src/layouts/Layout.astro`: - -```astro ---- -import { ViewTransitions } from "astro:transitions"; ---- - - - - - - - - - -``` - -This setup allows you to combine both backend and frontend flows for a page in Astro. Suppose you want to handle a form submission that includes the user's email, process it server-side, and display various outcomes based on your logic, all on the homepage. You can achieve this seamlessly within a single Astro route, like `src/pages/index.astro`: - -```astro ---- -const response = { form: false } - -if (Astro.request.method === 'POST') { - try { - // Indicate that the request is being processed - response.form = true - - // Get the user email from the form submissions - const data = await Astro.request.formData() - const userEmail = data.get('email') as string - - // Fetch feature flag from Xata - - // Define our own processing logic - } catch (e) { - // pass - } -} ---- - -{response.form && } - -
- - -
-``` - -### Fetch the feature flag object from Xata - -To customize the onboarding flow, the first step is to fetch the feature flag object from Xata. In this example, weā€™re gonna be using the `percentage` and `contains` properties of the feature flag to define our processing logic further. Using [Xata Read](https://xata.io/docs/sdk/get#getting-a-record-by-id), you can retrieve a record with a given ID, `the-flag` and destructure the relevant properties. - -```astro ---- -// Import the Xata Client created by Xata CLI in src/xata.js -import { XataClient } from '@/xata'; - -if (Astro.request.method === 'POST') { - // ... - // Fetch Feature from Xata - - // Create a Xata Client to fetch the flag from - const xata = new XataClient({ - apiKey: import.meta.env.XATA_API_KEY, - branch: import.meta.env.XATA_BRANCH - }); - - // Get the percentage property set in the flag - // Get the contains property set in the flag - const { percentage, contains } = await xata.db['my-flag-name'].read('the-flag'); - - // Define your processing logic - // ... -} ---- -``` - -### Implementing your own onboarding logic - -Based on the extracted values from the feature flag object, we'll onboard users whose emails end with `@someone.com`, as specified in the `contains` variable, only if the `percentage` value for this feature flag in Xata is over 30. By adjusting the `percentage` value in the \[Xata dashboard] to be above or below 30, you can effectively switch this feature flag on or off. - -```astro ---- -const response = { form: false, onboarded: false, userEmail: '' } - -if (Astro.request.method === 'POST') { - - // ... - // Define our processing logic - - if (userEmail) { - response.userEmail = userEmail - - // Check if email contains the expected, - // if yes onboard the user to a new flow - if (percentage > 30 && userEmail.endsWith(contains)) { - response.onboarded = true - } - - // If not, let's not onboard the user to the new flow - else { - response.onboarded = false - } - } - -} ---- - -{response.form && <>{response.onboarded ? : }} -``` - -### Deploy to Vercel - -The repository, is now ready to deploy to Vercel. Use the following steps to deploy: šŸ‘‡šŸ» - -1. Start by creating a GitHub repository containing your app's code. -2. Then, navigate to the Vercel Dashboard and create a **New Project**. -3. Link the new project to the GitHub repository you just created. -4. In **Settings**, update the _Environment Variables_ to match those in your local `.env` file. -5. Deploy! šŸš€ - -Way to go! You've now successfully added feature flags to you Astro app with Xata and Vercel, equipping your app for further development and success! - -### What's next?? - -We'd love to hear from you if you have any feedback on this tutorial, if you'd like to know more about Xata, or if you would like to contribute a community blog or tutorial. -Reach out to us on [Discord](https://xata.io/discord) or join us on [X | Twitter](https://twitter.com/xata). Happy building šŸ¦‹ diff --git a/file-attachments.mdx b/file-attachments.mdx deleted file mode 100644 index 2307f871..00000000 --- a/file-attachments.mdx +++ /dev/null @@ -1,180 +0,0 @@ ---- - title: 'File Attachments: Databases can now store files and images' - description: 'Extending a database record to include a new file column type with a global CDN, common security boundaries, and image transformations.' - image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/fa_blog_illustration.jpeg - alt: 'File Attachments: Databases can now store files and images' - author: Sorin Toma - date: 08-30-2023 - tags: ['engineering', 'file', 'attachments', 'fpFileAttachments'] - published: true - slug: file-attachments - ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/fa_blog_illustration.jpeg ---- - -Today, as part of [our launch week](https://xata.io/blog/launch-week-august-2023), weā€™re beyond excited to announce a feature that we've wanted to add ever since we started Xata: File Attachments. Think of it as having a new database column type where you can store files of any size, and behind the scenes they are stored in AWS S3 and cached through a global CDN. Files simply become a part of a database record. For example, they respect the same security boundary -- if you can access a record, you can also access its attached files. Image file types also get some extra functionality allowing you to request them at any size and style with built-in transformations. With this release, we aim to simplify your application architecture and reduce the number of services you need to manage. - -In this blog we'll dive into the capabilities released today and architecture behind our implementation. We hope you enjoy the read as much as we did building it šŸ˜ƒ - -### Adding file storage to the database - -Both files and relational data are ubiquitous in today's applications. Even the most basic scenarios generally require both structured relational data and file storage. Think about a blog which uses both posts metadata and images, a product catalogue or a document library. They all require basic data record management and query capabilities, as well as storage and access to large binary objects like images, videos, and documents. - -We often see engineers using a relational database with a separate storage service to store the data used by their application. In most cases the binary files are related to the relational data, therefore the common pattern is to store a file link in the database. This experience adds unnecessary friction and we saw an opportunity to simplify. Because file storage use cases are so closely correlated to data, we decided to embed [file attachments](/docs/sdk/file-attachments) directly into our serverless database and bring a new experience for building data apps that require binary storage. - -We've shown this feature to a number of developers before releasing it, and this is the type of feedback we've received on our approach so far: - -> Xata has become a critical part of my development cycle because I can quickly create and modify data structures without having to handle the database and migrations myself. Extending my records to include images has been great because I can easily apply transformations to them in the SDK. What I'm really excited about the ability to store any file type securely inside my database, it allows me to eliminate additional services like AWS S3.\ -> _Sebastian Gilits - Entrepreneur_ - -> We chose Xata as our DBaaS for KWARECOMā€™s Mapping and Reporting Portal because of its low latency and serverlessness in comparison to other options out there. Our performance expectations are met. Weā€™ve come to know and love their latest file storage offering; very unique and appealing. The CDN backed images are fast and the simplicity Xata provides means we at KWARECOM donā€™t need to worry about the complexity on the backend anymore.\ -> _Titus S. Zoneh - CEO @ KWARECOM_ - -> The DX for images and file storage in the Python SDK and REST API were out of this world. Iā€™m using images and file storage for an article classification tool. The simplicity of having image transformations baked into the record API is second to none. When my students at Berkeley are looking to spin up a quick project, Iā€™m going to point them to Xata.\ -> _Will Monge - Engineering Lead @ Good Research_ - -Letā€™s take a deeper look into what gets simplified and how Xata achieves this. - -### Attachments, not buckets - -Weā€™re really excited about today's release because it packs a lot of functionality into the simple experience of adding another column to your database. We wanted this experience to feel familiar, like adding and viewing an attachment in a spreadsheet instead of dropping a file into a generic bucket. When we started to work on the file attachments, we had the following goals in mind: - -1. Relational and storage APIs should share the same endpoints and the same connections. It is easier to work with and maintain one service rather than two. -2. APIs must share the same authorization scheme and the same permissions model. Avoid having to use different credentials and keep permissions in sync. -3. Relational data and binary object data should share the same region. Both data types should reside in the same compliance boundary and have similar guarantees. - -The first step in our design was to acknowledge that relational data and large binary objects have very different consumption models and trying to fit both into the same storage service leads to unacceptable compromises. One size doesnā€™t fit all. Serving large binary objects from a relational database cannot match the performance of a dedicated storage service in terms of compute cost, concurrency and throughput. The opposite is even more obvious, a storage service cannot match the querying and data management capabilities of a relational database. - -Like with any design challenge, we had to devise a solution to address seemingly conflicting requirements. The APIs for relational data and binary objects needed to be unified, while the backend storage had to differ to achieve the expected performance and feature set. - -### Introducing the file column type - -At the API and database schema level, the binary object data type became the `file` column type. You can attach one or multiple files in a column to a record in your database. This approach allowed all existing Xata features to work with the file type without any API change. - - - -Using the existing Xata REST APIs or SDKs, and the same connection, a developer can now upload a file, download a file, run queries over files using filters, aggregations, joins, and even run search queries to match file metadata. - -In the record model the file column holds a [JSON object](/docs/sdk/file-attachments#record-apis) with a predefined schema which contains both file metadata and the file content. - -```json -{ - "name": "Butterfree.png", - "mediaType": "image/png", - "size": 75, - "version": 1, - "attributes": { - "height": 475, - "width": 475 - }, - "base64Content": "iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAYAAABytg0kAAAAEklEQVR42mNk+M9QzwAEjDAGACCDAv8cI7IoAAAAAElFTkSuQmCC..." -} -``` - -#### Common read and write patterns for files - -Thinking further about typical relational data and file storage use cases, we noticed significant commonalities and differences. The management of files maps very well to relational data management. CRUD operations are similar in the sense that they work over organized data, entries have owners, there is a permission model in place, and they offer a level of consistency and durability. In general we can say that the write patterns are fairly similar. An application for managing files can very well use a database API to achieve the same. - -The important difference comes with the read patterns. Relational databases are designed for complex queries, which can involve high usage of CPU and memory, while storage reads have minimal CPU and memory usage. Because the operations are extremely simple, the storage reads generally scale to much higher request rate, concurrency and throughput. Therefore storage read patterns expect very high concurrency and throughput with minimal cost. Offering the database read for files retrieval will not meet the expectations for a storage service. - -We will call this high scale read use case, the content distribution scenario. To address content distribution, Xata introduces direct access URLs. They can be retrieved by reading the file column. Accessing the URL does not involve a database call and therefore they are NOT subject to Xata concurrency and rate limits, and can make use of the storage service capacity. - -### Access control for your files - -Since file attachments share the same endpoints with the Xata APIs, the same authorization scheme applies. Whether you are using API keys or OAuth, the same credentials can be used for managing files. This single service approach reduces complexity on the client side and at the same time guarantees that future authorization methods and future permissions models apply uniformly to both records API and file attachments API. - - - -As we discussed in the previous section, there are two distinct usage scenarios for files. The first is the common relational data approach where the operations are file CRUD and metadata query. The common authorization applies here. Access to the file is conditioned by the user having access to the database record. - -The second scenario is the content distribution through direct access URLs. In this case the authorization requirements are very different. If resource management requires a trusted security model, content distribution sometimes requires unrestricted access. - -To cater for different URL access needs, Xata provides 3 levels of authorization for the URLs that it generates. - -1. Public Access - the requests to retrieve the file are not subject to any authentication or authorization. This is very convenient for truly public content, but can be very dangerous if configured by mistake on sensitive data. By default, all uploaded files are private (the access URL requires authentication). Public access can be configured per file to allow maximum flexibility. The default can be updated per column, for scenarios where all files need to be public. -2. Signed URL - Xata can generate a signed URL which grants access to anyone having the URL for a specified amount of time. This is commonly used in scenarios where an image is rendered but the URL cannot be further shared because it expires shortly. The default timeout is 1 minute, but it can be configured per file. -3. Authenticated URL - the requests need to include a valid Authorization header in order to retrieve the file. - -All the access URLs offer lower latency compared to file retrieval through the Xata API because, apart from the authorization check (for signed and authenticated URLs) they go directly to storage skipping Xata middleware and the database service. - -### A built-in global content distribution network - -Going further on storage performance, modern applications require low latency across the globe. - -We could not claim to simplify working with files and then ask customers to configure and manage their own CDN to get reasonable global performance. As a result, the support for file attachments includes built-in CDN capabilities. There is no opt-in and no action is required to enable it; all direct access URLs are served through a CDN by default. In essence, all files retrieved through a URL are cached at the edge, making the following requests blazing fast. - -Xata opted to integrate [Cloudflareā€™s Global CDN](https://www.cloudflare.com/application-services/products/cdn/) for its wide geographical coverage, its performance and its feature set. - -As with any cache, the great performance improvement comes with the fundamental problem of stale cache entries and cache invalidation. Xata addresses stale cache entries by design using immutable file objects. This means any update to a file is in fact a new file object, with a different ID generating a different URL and eventually a different cache entry. This pattern is also known as _versioning_, because conceptually the cache keys change with every version of the object. - -Following this pattern, when the client application loads a set of records from Xata, it also gets the most up-to-date URLs which are guaranteed NOT to hit a stale cache entry, no matter where the cache is (browser, web proxy, CDN). This is very important because the user can clear the browser cache and Xata can invalidate the CDN cache, but a web proxy in between might still serve a stale cache entry. Versioning guarantees this can never happen. The downside is that file URLs are not persistent and they should not be used as static resources. However, this fits the Xata model of attaching binary objects to database records. Similar to the database records, the URLs are dynamic content and needs to be retrieved through database reads and queries. - -We have seen that content cannot be stale, but there is an important note on permissions. When a public object gets cached, changing permissions will not invalidate the cached entry. Making a file private applies immediately for new URLs, but the file is still accessible through old URLs until the cache entry expires in 2h. Xata advises greatest caution when configuring public access, because in practice there is a delay in changing the permissions from public to private. - -### Your images, your way with image transformations - -Looking at the most common scenarios where relational data is used together with binary files, the image use case stands out. All web applications use images today and images tend to require processing before they are rendered. Xata file attachments come with a [comprehensive set of image transformations](/docs/sdk/image-transformations) -- from resizing to photo adjustments to changing format and compression. - -All transformations are applied at the edge and are cached by default. Again, Xata leverages Cloudflare functionality for image transformations. - -We considered more flexible transformation definitions through query parameters or request body objects, but in the end chose the industry standard of defining the transformations in the URL path. This makes it easier to embed the transformation into a web page and the transformation is automatically included in the cache key within the CDN. - -`https://eu-west-1.storage.xata.sh/transform/rotate=180,height=50/nj42n37o4l3dd19fe6vsh4plkk` - -### Building a file attachment service - -Generally, our philosophy is to abstract away complexity where we can so our end users donā€™t have to worry about it. File attachments is no exception to this strategy. - -![File attachments architecture](/images/fa_architecture.png) - -Behind the scenes, the file data is actually stored in two places. The file content is stored as an AWS S3 object while the file metadata (name, type, size, S3 pointer) is stored as a JSON object in the PostgreSQL database table. - -We chose S3 for the binary object storage for its high performance, durability, availability and because it shares the same compliance certifications with the AWS Aurora, which we use for running PostgreSQL. This way, all data (relational and binary objects) is located in the same data centers and benefits from the same compliance guarantees. - -The fundamental challenge when writing state to two different services is making it transactional. Xata service implements two-phase commit semantics to ensure that a file write either completes successfully or is rolled back. This is one of the key operations where Xata does the heavy lifting and abstracts away the complexity. The client code can rely on the transactional guarantee and no longer be concerned with two-phase commit or dealing with an out of sync state. - -A second set of challenges comes around data deletion and cleanup, where again it is not trivial to keep the state in sync between two services. With database tables, Xata takes the cautious approach of delayed cleanup. This allows us to offer undo delete operations (not exposed yet) and to have a general recovery option in case of accidental deletes. When file content storage comes into the picture, it needs to follow the same pattern. Restoring only half of the deleted data is not particularly useful. - -The implemented solution treats the PostgreSQL metadata as source of truth and removes the associated file data only when the corresponding records are deleted from PostgreSQL. This is achieved by hooking into the PostgreSQL replication and handling delete record events. This is an elegant way of replicating the deleted state from PostgreSQL to S3, but unfortunately it is only half the solution, because it only handles individual record or value deletes. When data is deleted by dropping entire columns, tables, databases the events are not captured in replication so Xata handles this separately by scheduling bulk deletes. - -Backup support adds another level of complexity. Xata creates regular backups and can perform a restore on request. The binary object storage needs to match the ability to restore the state from the time of the database backup. This is achieved through a combination of immutable S3 objects and configuring S3 lifecycle. Deleted files become inaccessible, but they are kept for 7 days after their deletion for recovery purposes. After the 7 days the files are permanently deleted. - -### Additional API considerations - -If you look at the file column JSON example shown earlier, it might strike you as odd. Files donā€™t come encoded as base64 and they are certainly not used in their encoded form. - -Xata APIs are JSON based and therefore the file content needed to fit in a JSON object. We chose to use Base64 encoding as it is the most common binary encoding and has the widest library support across languages. For a client application that has binary content in a buffer, it should be trivial to encode it as Base64. The Xata SDK offers helpers [here](https://xata.io/docs/sdk/file-attachments). - -While we recommend this approach when dealing with small files and when extending existing apps that already use Xata APIs, we acknowledge there are drawbacks in using the Base64 encoding. For very large files or for very high throughput applications, the extra CPU cost of encoding and decoding the files begins to matter and it is exposed in the delivery latency. Also, the extra 33% in file size added by the encoding impacts the size on the wire and implicitly the performance and bandwidth costs. - -To alleviate these concerns we introduced [binary file APIs](/docs/sdk/file-attachments). These APIs use similar endpoints but they accept and they retrieve binary content. For files larger than 20MB these are the only APIs for uploading content. - -It is important to mention these are still database APIs, they involve reads and writes of metadata to PostgreSQL and they are not direct S3 wrappers. - -### Example image gallery app - -We have launched this functionality with an example image gallery app to highlight the developer experience you get with Xata and give you a starting point to try it out. This example combines full-text search, aggregations, files, and image transformations. If youā€™d like to get started, be sure to check out our [example repository](https://github.com/xataio/sample-nextjs-chakra-gallery-app). - -![Gallery example with file attachments](/images/fa_example_gallery_app.png) - -### Shout out to the Xata community - -Before releasing file attachments, we decided to open up an early access program for those interested in our community. It was our first early access program since our private beta last year, and we were blown away by the amount of support and engagement we received from our honorary Xataflies. We sincerely want to thank everyone that participated. This feature would not be as amazing as it is today without your help šŸ™ - -Want to join in on the fun with our community? Sign up for our content hackathon [here](https://xata.io/blog/launch-week-august-2023) to win some prizes and get some sweet Xata swag. - -### Get started today - -Xata is simplifying the patterns of working with relational data and binary objects together. If you are facing any of the problems described in this post, we welcome you to [sign up](https://app.xata.io/) and try Xata. Weā€™d love your feedback on this feature, if you have any suggestions, questions, or issues reach out to us on [Discord](https://xata.io/discord) or follow us on [X / Twitter](https://twitter.com/xata). diff --git a/indexing-files-with-python.mdx b/indexing-files-with-python.mdx deleted file mode 100644 index 0cbe01df..00000000 --- a/indexing-files-with-python.mdx +++ /dev/null @@ -1,452 +0,0 @@ ---- - title: 'Indexing File Attachments with the Python SDK' - description: 'Extracting text from File Attachments and indexing it with Python.' - image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/index-file-attachments.jpg - alt: 'Extracting text from File Attachments and indexing it with Python.' - author: Kostas Botsas - date: 09-14-2023 - tags: ['file', 'attachments', 'fpFileAttachments'] - published: true - slug: index-files-with-python - ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/index-file-attachments.jpg ---- - - -The all-new [File Attachments](https://xata.io/docs/sdk/file-attachments) column types enable storing files in your database alongside other common [column data types](https://xata.io/docs/concepts/data-model#column-types). Once uploaded, the attachment lives in Xataā€™s object store and it can be reached as a column value with additional metadata such as `mediaType`, `size` and `attributes`, which can be used in queries, filters and summaries. Xataā€™s API provides several methods for [File Access](https://xata.io/docs/sdk/file-attachments#file-access-urls), including [Authenticated](https://xata.io/docs/sdk/file-attachments#authenticated-urls), [Public](https://xata.io/docs/sdk/file-attachments#public-urls) and [Signed](https://xata.io/docs/sdk/file-attachments#signed-urls) URLs. - -The actual file content is not indexed by default, though. Indexing refers to the process of extracting a fileā€™s text content and storing it under a text column, where it can be queried and searched. How can we use Xataā€™s advanced features such as [free-text search](https://xata.io/docs/sdk/search) and [Ask AI](https://xata.io/docs/sdk/ask) on our file attachmentā€™s actual content? - -In this article we are exploring methods for indexing File Attachments using Xataā€™s Python SDK, which is now in [General Availability](https://xata.io/blog/announcing-the-python-sdk-ga). - -## Overview - -The `xfileindex` script in the [xtools](https://github.com/xataio/xtools) repo can be used to extract content from text or PDF file attachments stored in Xata and indexed as raw text, which allows us to use the content with query, free-text search, and Ask AI capabilities. - -```bash -python xfileindex.py --db https://{workspace}.{region}.xata.sh/db/{db} --table mytable --columns myfile,mymultiplefiles -``` - -The [README](https://github.com/xataio/xtools/blob/main/xfileindex/README.md) file provides usage examples as well as the full list of parameters. - -## Using Xataā€™s Python SDK - -Python is a programming language that comes pre-installed on many popular OS distributions, but it can also be [downloaded](https://www.python.org/downloads/) and installed on any platform. The Python package installer, [pip](https://pip.pypa.io/en/stable/getting-started/), almost always exists alongside Python wherever it is installed (but there are other ways to [install pip](https://pip.pypa.io/en/stable/installation/)). Pip fetches modules from the official [Python Package Index(PyPI)](https://pypi.org), where you can find the [Xata Python SDK](https://pypi.org/project/xata/) package. It can be installed with the command `pip install xata` - check the [documentation](https://xata.io/docs/sdk/python/overview) for more details. - -With the Xata Python SDK installed, we can import the `XataClient` in our code (`from xata.client import XataClient`) and interact with our Xata database. - -Note that the Python version in your system (`python --version`) should be 3.8 or higher, in order to use the [script](https://github.com/xataio/xtools/tree/xfileindex-v0.0.1/xfileindex). - -## Python code to index File Attachments - -Now that weā€™ve covered how to use Xata with Python, letā€™s jump straight to the point of how to index file attachments. - -We will assume that youā€™ve already uploaded some files in your database under the `file` or `file[]` column type. Check out the [File Attachments documentation](https://xata.io/docs/sdk/file-attachments) for more details on those column types. - -Our [script](https://github.com/xataio/xtools/tree/main/xfileindex) scrolls through the Xata database, looks for `file` or `file[]` columns with text, CSV or PDF content (*mediaType*), extracts the text and writes it into text columns in a target table. This is all you need to put the file content to work with query, search, and Ask AI features! - -Here are the essential steps for running this script: - -1. Clone the xtools repo: - ```bash - git clone https://github.com/xataio/xtools - ``` - This repo is a library of open source helper scripts provided by Xata, but we welcome community contributions. -2. Navigate to the directory: - ```bash - cd xtools/xfileindex - ``` -3. Install the required packages using pip: - ```bash - pip install -r requirements.txt - ``` - The script uses some extra packages (xata, PyPDF2) which are not typically available with the default Python installed in your system, so we need to make sure theyā€™re installed before we proceed. -4. Set your Xata API key as an environment variable on your terminal. You can generate an API key in the [account settings](https://app.xata.io/settings) page. Alternatively, you can use [a `.env` file](/docs/sdk/python/overview#dotenv), to store the API key. - ```bash - export XATA_API_KEY="xau_mykeyā€ - ``` -5. We need to gather a few parameters about our database and schema: - - The database endpoint URL. You can find that on the Settings page of your database, under ā€œDatabase endpointā€. - - The table name where our files are stored. - - The name(s) of the column(s) where your files are stored. -6. Knowing all of the above, you can run the script as follows: - - ```bash - python xfileindex.py --db https://{workspace}.{region}.xata.sh/db/{db} --table tablename --columns firstcolumn,secondcolumn - ``` - - - - ```bash - python xfileindex.py --db https://ws-fdrujb.eu-central-1.xata.sh/db/myfilesdb --table mytable --columns firstcolumn,secondcolumn - Created new table mytableIndex - - Downloading file from table: mytable , record: rec_cjmvbvr9o1hng2274a9g , column: firstcolumn , filename: lotr.pdf - - Processing record: rec_cjmvbvr9o1hng2274a9g , column: firstcolumn , filename: lotr.pdf , type: application/pdf in 1199 chunks: - Indexed 100 / 1199 chunks. - Indexed 200 / 1199 chunks. - Indexed 300 / 1199 chunks. - Indexed 400 / 1199 chunks. - Indexed 500 / 1199 chunks. - Indexed 600 / 1199 chunks. - Indexed 700 / 1199 chunks. - Indexed 800 / 1199 chunks. - Indexed 900 / 1199 chunks. - Indexed 1000 / 1199 chunks. - Indexed 1100 / 1199 chunks. - Indexed 1199 / 1199 chunks. - - Downloading file from table: mytable , record: rec_cjmscqj9n6h19i0nbdng , column: secondcolumn , filename: sample-2mb-text-file.txt - - Processing record: rec_cjmscqj9n6h19i0nbdng , column: secondcolumn , filename: sample-2mb-text-file.txt , type: text/plain in 11 chunks: - Indexed 11 / 11 chunks. - - Downloading file from table: mytable , record: rec_cjmscqj9n6h19i0nbdng , column: secondcolumn , filename: mycsv.csv - - Processing record: rec_cjmscqj9n6h19i0nbdng , column: secondcolumn , filename: mycsv.csv , type: text/csv in 1 chunk: - Indexed 1 / 1 chunk. - ``` - - -### Under the hood - -Letā€™s take a moment to review the output and dive deeper into how things work. - -The script creates a new table `mytableIndex`, by appending "Index" to the source table name. Alternatively you can specify your preferred name for the target table by providing the `dest` parameter (`--dest customTableName`). Also, in case your table lives in a branch other than `main`, you can add the branch parameter (`--branch dev`). - -The script opens a ā€œscrollā€, a [cursor-based](/docs/sdk/get#cursor-based-pagination) query to the table, which goes through records looking for content under the specified columns (`firstcolumn`, `secondcolumn`). An intermediate function `ensure_target_table` makes sure the target table schema is compatible with the output that is going to be generated by the script, but you wonā€™t typically be concerned about that - unless you want to customize the schema of the resulting table, in which case feel free to dive into the relevant [code section](https://github.com/xataio/xtools/blob/xfileindex-v0.0.1/xfileindex/xfileindex.py#L234). - -Here is how Xata's ā€œscrollā€ works in Python: - -```python -xata = XataClient(db_url=f"{TARGET_DB}:{BRANCH}") -querypayload = {"columns": COLUMNS_TO_INDEX, "page": {"size": PAGE_SIZE}} -more = True -while more: - response = xata.data().query(SOURCE_TABLE, querypayload) - if response.is_success(): - process_response(xata, response) - more = response.has_more_results() - if more: - page = {"after": response.get_cursor(), "size": PAGE_SIZE} - querypayload = {"columns": COLUMNS_TO_INDEX, "page": page} -``` - -An overview of the variables in use include: - -* **`COLUMNS_TO_INDEX`**: an array with the content of the `--columns` input parameter. - -* **`PAGE_SIZE`**: the number of records to fetch from Xata at once with every scroll iteration. The query does not fetch the file content at this stage, just metadata informing you about the file and its format. Using the maximum page size of 200, it fetches as many records as possible in a single round trip which is usually the fastest approach. - -* **`SOURCE_TABLE`**: the content of the `--table` input parameter. - -Alongside the actual records, you also get: - -- a flag that you can check with `has_more_results()` which informs you whether there are more records matching your query (containing content in the specified columns). -- a cursor that we retrieve with `get_cursor()` which points the following query to the next page of results (if any). - -## Extracting text from attachments - -Text extraction takes place in the function `process_response(xata, response)`. - -As shown earlier, the scroll query fetches records containing file metadata, but not the file _content_. After you find a record and column where a file lives, you can proceed to download the fileā€™s *content*. - -You can determine whether the file lives in a single file or file array column by retrieving the table schema and looking into the column type defined in it. Just to keep things simple we can also extrapolate this based on whether the column response is a dictionary (single file) or an array (hence, potentially multiple files): - -```python -if column in record and type(record[column]) == dict: - column_type = "single_file" -elif column in record and type(record[column]) == list: - column_type = "multiple_files" -``` - -Depending on whether the column you're working with is a `file` or `file[]`, you download the file content using the `xata.files().get` or `xata.files.get_item()` call respectively: - -```python -if column_type == "single_file": - file = xata.files().get( - SOURCE_TABLE, - record["id"], - column - ) -elif column_type == "multiple_files": - file = xata.files().get_item( - SOURCE_TABLE, - record["id"], - column, - column_file["id"] - ) -``` - -The difference between the two methods is that content from single files (`.get()`) is retrieved using the combination of record ID and column name, while content from file arrays (`.get_item()`) additionally requires the ID of the file - since files stored in an array column also get a file ID (automatically or explicitly assigned) so they can be identified in the array. - -File column metadata include the `mediaType` attribute, which informs you of the fileā€™s type. Use this information to process content accordingly for plain text, CSV or PDF files: - -```python -def process_file(file, mediaType): - mediaType = mediaType.lower() - if mediaType.startswith("text/plain") or mediaType.startswith("text/csv"): - chunked_text = process_text_file(file) - elif mediaType.startswith("application/pdf"): - chunked_text = process_pdf_file(file) - else: - chunked_text = [] - return chunked_text -``` - -> NOTE: CSV files are treated as plain text in the context of this script. We recommend using the purpose-built iImport CSV](/docs/csv-data/import-data) feature for ingesting CSV files in a schema that mirrors the CSV file structure. - -Text files are split to chunks in order to fit into Xataā€™s [text](https://xata.io/docs/concepts/data-model#text) column type, which can store a maximum payload size of 200KB as stated in the [limits](/docs/rest-api/limits#column-limits) page. The default value of the optional `maxchunk` parameter, which sets the `MAX_TEXT_COLUMN_LENGTH` variable, is 200000 (characters) to satisfy this requirement. - -The `process_text_file` method then calls the wrap function from the [textwrap](https://docs.python.org/3/library/textwrap.html) module, to create chunked text. The result is a list of paragraphs with the requested max width. - -```python -def process_text_file(file): - chunked_text = wrap( - str(file.content.decode(ENCODING)), - width=MAX_TEXT_COLUMN_LENGTH, - drop_whitespace=False, - break_on_hyphens=False, - expand_tabs=False, - replace_whitespace=False, - ) - return chunked_text -``` - -The `process_pdf_file` method calls the PdfReader function from the [PyPDF2](https://pypi.org/project/PyPDF2/) module to extract text from each page of the PDF. Additionally, in case a page contains more text than specified by the `maxchunk` parameter, it applies the same text processing wrap function which may split the PDF page to more chunks if necessary. - -```python -def process_pdf_file(file): - with BytesIO(file.content) as open_pdf_file: - reader = PdfReader(open_pdf_file) - chunked_text = [] - for page_iterator in range(len(reader.pages)): - pdf_page = reader.pages[page_iterator] - extracted_text = pdf_page.extract_text() - chunks = wrap( - str(extracted_text), - width=MAX_TEXT_COLUMN_LENGTH, - drop_whitespace=False, - break_on_hyphens=False, - expand_tabs=False, - replace_whitespace=False, - ) - chunked_text.extend(chunks) - return chunked_text -``` - -Similar functions can be implemented for other mediaTypes to support more file types, but that may require installing additional libraries and software (such as [Pandoc](https://pandoc.org/)). - -## Indexing text chunks into Xata - -Now that you have extracted text from the file attachment into a `chunked_text` array containing chunks of appropriate max length, it is time to ingest them into Xata. The `ingest_chunks` method does just that, while offering a few options regarding our approach for overwriting records and ingesting content in bulks. - -### Automatically generated vs deterministic record IDs - -In other words: do we prefer to overwrite text content for the same files if it already exists (i.e. when re-running the script), or do we always append new records? - -Every record in a table is uniquely identified by the built-in [id column](https://xata.io/docs/concepts/data-model#id). By specifying a deterministic id pattern you can overwrite records if they already exist. Otherwise if you donā€™t specify any ID for a record, Xata automatically assigns one, which means you write a new record every time you index a chunk of text, potentially creating duplicates if the script runs multiple times on the same tables. The preferred approach can be specified with the `--id` input parameter, providing a value between `deterministic` or `random`. - -With the deterministic approach, record IDs are generated by concatenating the origin recordā€™s ID, the source table name, the source column name and an iterator for each chunk of text from this file. Additionally, for `file[]` columns, include the ID of the files in the array. - -```python -if ID_STRATEGY == "deterministic": - if column_type == "single_file": - chunk_rec_id = ( - f'{source_record["id"]}-{SOURCE_TABLE}-{column}-{chunk_iterator}' - ) - elif column_type == "multiple_files": - chunk_rec_id = f'{source_record["id"]}-{SOURCE_TABLE}-{column}-{column_file["id"]}-{chunk_iterator}' -``` - -The deterministic ID approach requires the use of the [upsert](/docs/sdk/update#replacing-a-record) method, or the [update transaction operation](https://xata.io/docs/sdk/transaction#updates) with the upsert parameter set to `true`, to replace existing records. - -With the non-deterministic (random) approach, you simply do not include any ID when writing to Xata with the [insert](https://xata.io/docs/sdk/insert) method or [insert transaction operation](https://xata.io/docs/sdk/transaction#inserts), which automatically assigns a new ID to the created record. - -### Atomic writes vs Transactions - -The `mode` parameter allows for selection between the `atomic` or `transaction` indexing approach. -In both cases when emitting text content to Xata the response code should be checked for the retriable status code `429`, returned in case you hit the branch [rate limits](/docs/rest-api/limits#rate-limits). - -ā€œAtomicā€ means writing one record, a single chunk of text, with each HTTP/S request to Xata. - -- Atomic requests with deterministic record IDs use the [upsert](https://xata.io/docs/sdk/update#replacing-a-record) method: - -```python -if ID_STRATEGY == "deterministic": - #... - if MODE == "atomic": - resp = xata.records().upsert(TARGET_TABLE, chunk_rec_id, content_record) - if resp.is_success(): - print( - " id:", - chunk_rec_id, - "size:", - len(content_record["content"]), - "chars", - ) - else: - print("Response", resp.status_code, resp) - while resp.status_code == 429: - print("Throttled. Retrying...") - resp = xata.records().upsert(TARGET_TABLE, chunk_rec_id, content_record) -``` - -- Atomic requests with autogenerated record IDs use the [insert](/docs/sdk/insert) method: - -```python -elif ID_STRATEGY == "random": - #... - if MODE == "atomic": - resp = xata.records().insert(TARGET_TABLE, content_record) - if resp.status_code == 201: - print( - " id:", - resp["id"], - "size:", - len(content_record["content"]), - "chars", - ) - else: - print("Response", resp.status_code, resp) - while resp.status_code == 429: - print("Throttled. Retrying...") - resp = xata.records().insert(TARGET_TABLE, content_record) -``` - -Atomic writes are useful for low volume use cases and for troubleshooting purposes, as in case of a write error we would receive a characteristic response code and error context about the erroring part of the request. - -However, this approach does not usually perform well at scale as it requires the client to initiate a network connection and add HTTP/S protocol overhead (headers, encryption) for, comparably, little payload. - -By including multiple chunks of text in a single write request, we can achieve better ā€œpayload to overheadā€ ratio and our code ultimately works faster. This can be done with the [bulk_insert](/docs/sdk/insert#creating-records-in-bulk) method or with [transaction insert](/docs/sdk/transaction#inserts) operations. - -A distinctive difference between the two approaches for this particular case, is that transactions can also perform multiple upsert (replace record) operations at once, while the bulk endpoint only supports creating multiple *new* records, so it can be used only with the autogenerated ID approach. - -All operations within a transaction request are rolled back (aborted) in case any operation fails. - -Performance-wise, both methods would be very close. Although we could use bulks for autogenerated ids, just for uniformity we use the Transaction method for both deterministic and autogenerated IDs. - -The Transaction helper method handles the grouping of transaction operations. - -```python -if MODE == "transaction": - trx = Transaction(xata) -``` - -- Appending an ā€œupdateā€ operation with deterministic record IDs using the **upsert** parameter (set to `True`): - -```python -if ID_STRATEGY == "deterministic": - #... - elif MODE == "transaction": - trx.update(TARGET_TABLE, chunk_rec_id, content_record, True) -``` - -- Appending an ā€œinsertā€ operation with random (autogenerated) record IDs: - -```python -elif ID_STRATEGY == "random": - #... - elif MODE == "transaction": - trx.insert(TARGET_TABLE, content_record) -``` - -Once we reach the maximum number of operations that can run as a single transaction, as set by the `tsize` script parameter, we call the `trx.run()` method. - -```python -resp = trx.run() -if resp["status_code"] == 200: - # print success -else: - # error handling - while resp["status_code"] == 429: - print("Throttled. Retrying...") - trx.operations = retriable_operations - resp = trx.run() -``` - -### Putting the indexed data to work - -Now that youā€™ve indexed the content of file attachments to Xata, you can unleash the full potential of advanced [free-text search](/docs/sdk/search) and [Ask AI](https://xata.io/docs/sdk/ask) capabilities. - -As hinted in the script output above, one of the file attachments in the records indexed, was The Lord of the Rings (lotr.pdf). - -Jumping to the Playground section in the Xata Web UI, you can ask AI questions to the generated table `mytableIndex`: - -```ts -import { getXataClient } from "./xata"; -const xata = getXataClient(); - -const response = await xata.db.mytableIndex.ask("Why was the One Ring forged?", { - rules: ["Only answer questions using the provided context."], -}); - -console.log(response); -``` - -Response: - -```json -{ - "answer": "The One Ring was forged by Sauron to be his master and to control the other Rings of Power.", - "sessionId": "eh6adiosf10ct97qfahv70kn2k", - "records": [ - "rec_cjobbv3l4agk1ks239dg-mytable-firstcolumn-15", - "rec_cjobbv3l4agk1ks239dg-mytable-firstcolumn-517", - "rec_cjobbv3l4agk1ks239dg-mytable-firstcolumn-263" - ] -} -``` - -Go ahead and give it a try - you can upload this [example file](https://us-east-1.storage.xata.sh/qdbf929rfh0uvajknm9slm00v4) to your table and index it with this script: - -```bash -python xfileindex.py --db https://ws-fdrujb.eu-central-1.xata.sh/db/myfilesdb --table mytable --columns myfiles -Created new table mytableIndex - -Downloading file from table: mytable , record: rec_cjob8i3l4agk1ks239a0 , column: myfiles , filename: jokes.pdf -- Processing record: rec_cjob8i3l4agk1ks239a0 , column: myfiles , filename: jokes.pdf , type: application/pdf in 1 chunk: - Indexed 1 / 1 chunk. -``` - -The result is one record with a single chunk of text since the attachment is a one-page pdf with text content that fits within the character limit of a chunk (200000 characters). - -In the Playground, we can ask questions on the fileā€™s content: - -```ts -import { getXataClient } from "./xata"; -const xata = getXataClient(); - -const response = await xata.db.mytableIndex.ask("Why do I need a cat?", - { - rules: [ - "Only answer questions using the provided context." - ] - }); - -console.log(response); -``` - -Response: - -```json -{ - "answer": "You need a cat because it will remind you that you don't deserve unconditional love.", - "sessionId": "53nseq27f905pev4lhs4duiv3k", - "records": [ - "rec_cjob8i3l4agk1ks239a0-mytable-myfiles-0" - ] -} -``` - -Time to upload your own text, csv and pdf files to Xata and get them indexed easily! - -## Indexing local files to Xata - -If you are looking to index some of your locally stored files directly into Xata without storing them as attachments first, we got you covered: [localfileindex](https://github.com/xataio/xtools/blob/xfileindex-v0.0.1/xfileindex/localfileindex.py) is another flavor of this script, it extracts text from your local files and indexes the contents to Xata. - -Check out the [readme](https://github.com/xataio/xtools/tree/main/xfileindex#localfileindex) for usage instructions. - -## Conclusion - -Xata is committed to making data easy to work with. We will keep adding new features and enriching the file attachment column types and the SDKs. - -If you have feedback or questions, you can reach out to us on [Discord](https://xata.io/discord), [X / Twitter](https://twitter.com/xata) or submit a [support request](https://support.xata.io/hc/en-us/requests/new). diff --git a/json-type.mdx b/json-type.mdx deleted file mode 100644 index 41ee6bf8..00000000 --- a/json-type.mdx +++ /dev/null @@ -1,319 +0,0 @@ ---- -title: 'Introducing the JSON column type' -description: 'Use the new JSON column type for improved data storage in Xata.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/json-type/json-column.jpeg - alt: JSON type header -author: Alejandro MartĆ­nez -date: 09-01-2023 -tags: ['engineering'] -published: true -slug: json-column-type -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/json-type/json-column-og.jpeg ---- - -Data models that use schemas are great. At Xata, we believe they're a solid choice in most scenarios. But, we also know that not every piece of data fits perfectly into the relational model or is not as convenient as schemaless, and sometimes, especially in the early stages of a project, you want something more flexible and straightforward. That's where using JSON documents within a relational data store comes in handy. It offers the best of both worlds ā€“ structure when you need it and a bit of freedom when you don't. - -Many of our users have been [asking for this feature](https://xata.canny.io/feature-requests/p/json-objects), and as part of [launch week](/blog/launch-week-august-2023) we are happy to announce that it's finally here. - -Basic support for [JSON column type](https://xata.io/docs/sdk/filtering#json) has been added and it will bring many benefits including: - -- **Enhanced flexibility** by providing a way to store schemaless data in a relational database -- **Data integrity** with JSON validation according to [RFC 7159](https://www.rfc-editor.org/rfc/rfc7159.html) -- **Streamlined development** that stores any unstructured data directly with no need to handle a schema or data conversions -- **Efficient queries** so you can apply many filters to nested nodes -- **Search functionality**, as JSON documents are indexed as any other Xata data type and can be searched using the [full-text search](https://xata.io/docs/sdk/search) capabilities of Xata - -We plan on extending Xata support for JSON even further in the future, but the current capabilities are already very powerful and will solve most use cases. -Below we provide examples of how you can start using JSON documents in Xata today. - -Let's think a bit about a simple data model for an online shop. Suppose we have a _Products_ table. - -![Products table](/images/json-type/products.png) - -## Creating a JSON column - -Sometimes different categories of products have completely different specs, so we don't want to create a column for each of them. -We can use a JSON column to store the product details. Let's do this by adding a `details` field to our table. You can do this via the UI or via the API like this: - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/tables/{table}/columns - -{ - "name": "details", - "type": "json" -} -``` - -Let's add a few products with different details, for instance: - -- A t-shirt with size and color -- A book with author, ISBN and number of pages -- A climbing rope with a length and a thickness - - -```ts -const record1 = await xata.db.Products.create({ - name: 'Xata xwag T-shirt', - details: { - color: 'purple', - size: 'M', - } -}); -const record2 = await xata.db.Products.create({ - name: 'Meditations', - details: { - author: 'Marcus Aurelius', - isbn: '978-0140449334', - pages: 304 - } -}); -const record3 = await xata.db.Products.create({ - name: 'Long climbing rope', - details: { - length: 80, - thickness: 9.8, - color: 'blue', - } -}); -``` - -```py -record1 = xata.records().insert("Products", { - "name": "Xata xwag T-shirt", - "details": { - "color": "purple", - "size": "M", - } -}) -record2 = xata.records().insert("Products", { - "name": "Meditations", - "details": { - "author": "Marcus Aurelius", - "isbn": "978-0140449334", - "pages": 304 - } -}) -record3 = xata.records().insert("Products", { - "name": "Long climbing rope", - "details": { - "length": 80, - "thickness": 9.8, - "color": "blue" - } -}) -``` - -```jsonc - // Not yet available -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/tables/{table}/data -{ - "name": "Xata xwag T-shirt", - "details": "{\"color\": \"purple\", \"size\": \"M\"}" -} - -{ - "name": "Meditations", - "details": "{\"author\": \"Marcus Aurelius\", \"isbn\": \"978-0140449334\", \"pages\": 304}" -} - -{ - "name":"Long climbing rope", - "details":"{\"length\": 80, \"thickness\": 9.8, \"color\": \"blue\"}" -} -``` - - - -It's important to note that the JSON documents are processed and stored in a binary format in order to improve querying and storage performance. -This has the following implications: - -- White spaces are not preserved -- Key order is not preserved -- In case of duplicate key, only the last one is stored - -## Querying JSON documents - -### The arrow notation `->` - -This is PostgreSQL's syntax for navigating JSON fields. It's used to access the value of any JSON node, no matter how deep in the tree. -Xata uses a similar notation to query data and apply some of the existing filters to any JSON value. PostgreSQL uses different operators and casting -depending on the data types, but Xata is able to infer the data type from the provided value and apply the correct operator. -So far, comparison by strings and numbers is supported but this will be extended in the near future. - -### Filter products size 'M' - - -```ts -const records = await xata.db.Products.filter({ - "details->size": 'M' -}).getMany(); -``` - -```python -records = xata.data().query("Products", { - "filter": { - "details->size": "M" - } -}) -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/sql - -{ - "statement": "SELECT * FROM \"Products\" WHERE details->>'size' = 'M';" -} -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/tables/{table}/query - -{ - "filter": { - "details->size": "M" - } -} -``` - - - -### Filter products with a length greater than 50 meters - - -```ts -const records = await xata.db.Products.filter({ - "details->length": { - "$gt": 50 - } -}).getMany(); -``` - -```python -records = xata.data().query("Products", { - "filter": { - "details->length": { - "$gt": 50 - } - } -}) -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/sql - -{ - "statement": "SELECT * FROM \"Products\" WHERE (details->>length)::numeric > 50;" -} -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/tables/{table}/query - -{ - "filter": { - "details->length": { - "$gt": 50 - } - } -} -``` - - - -### Check for a substring in a nested JSON node - - -```ts -const records = await xata.db.Products.filter({ - "details->author": { - "$contains": "Marcus" - } -}).getMany(); -``` - -```python -records = xata.data().query("Products", { - "filter": { - "details->author": { - "contains": "Marcus" - } - } -}) -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/sql -{ - "statement": "SELECT * FROM \"Products\" WHERE details->>author LIKE '%Marcus%';" -} -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/tables/{table}/query -{ - "filter": { - "details->author": { - "$contains": "Marcus" - } - } -} -``` - - -### Other general control or negation operators work as well - - -```ts -const records = await xata.db.Products.filter({ - "$not": { - "details->length": { - "$gt": 50 - } - } -}).getMany(); -``` - -```python -records = xata.data().query("Products", { - "filter": { - "$not": { - "details->length": { - "$gt": 50 - } - } - } -}) -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/sql -{ - "statement": "SELECT * FROM \"Products\" WHERE NOT (details->>length)::numeric > 50;" -} -``` - -```jsonc -// POST https://{workspace}.{region}.xata.sh/db/{db}:{branch}/tables/{table}/query -{ - "filter": { - "$not": { - "details->length": { - "$gt": 50 - } - } - } -} -``` - - - -## Conclusion - -Xata is committed to simplifying the way you work with data. We will keep improving our offering by both adding more rich data types and extending the capabilities of the current ones. -Basic JSON support is one more step in that direction along with the previously released [files attachments](https://xata.io/blog/file-attachments). - -If you have feedback or questions, you can reach out to us on [Discord](https://xata.io/discord) or [X / Twitter](https://twitter.com/xata). diff --git a/keyword-vs-semantic-search-chatgpt.mdx b/keyword-vs-semantic-search-chatgpt.mdx deleted file mode 100644 index 2099ea05..00000000 --- a/keyword-vs-semantic-search-chatgpt.mdx +++ /dev/null @@ -1,409 +0,0 @@ ---- -title: 'Semantic or keyword search for finding ChatGPT context. Who searched it better?' -description: 'The blog post compares keyword search with semantic/vector search for the task of selecting context for an ChatGPT-based questions and answers bot.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/how-to-fuzzy-search-with-xata.png - alt: Keyword versus vector search -author: Tudor Golubenco -date: 03-06-2023 -tags: ['engineering', 'search', 'ai', 'fpGenerativeAi'] -published: true -slug: keyword-vs-semantic-search-chatgpt ---- - -> Hey there, you are on the Xata engineering blog. Xata is a serverless data platform on top of PostgreSQL, offering both keyword and semantic search based on Elasticsearch. [Sign up today](https://xata.io/)! - -Last week weā€™ve added a Q\&A bot that answers questions from -[our documentation](https://xata.io/docs/overview?show-chat). This leverages -the ChatGPT tech to answer questions from the Xata documentation, even though -the OpenAI GPT model was never trained on the Xata docs. - - - -The way we do this is by using an approach suggested by Simon Willison -in this [blog post](https://simonwillison.net/2023/Jan/13/semantic-search-answers/). -The same approach can be found also in an -[OpenAI cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb). -The idea is the following: - -- Run a text search against the documentation to find - the content that is most relevant to the question asked by the user. -- Produce a prompt with this general form: - -```txt -With these rules: {rules} -And this text: {context} -Given the above text, answer the question: {question} -Answer: -``` - -- Send the prompt to the ChatGPT API and let the model complete the answer. - -We found out that this works quite well and, combined with a relatively -low model temperature (the concept of temperature is explained in this -[blog post](https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/)), -this tends to produce correct results and code snippets, as long as the answer can -be found in the documentation. - -A key limitation to this approach is that the prompt that you build in -the second step above needs to have max 4000 tokens (~3000 words). This -means that the first step, the text search to select the most relevant -documents, becomes really important. If the search step does a good -job and provides the right context, ChatGPT tends to also do a good -job in producing a correct and to-the-point result. - -So whatā€™s the best way to find the most relevant pieces of content in the -documentation? The OpenAI cookbook, as well as Simonā€™s blog, use what -is called semantic search. Semantic search leverages the language model -to generate embeddings for both the question and the content. Embeddings -are arrays of numbers that represent the text on a number of dimension. -Pieces of text that have similar embeddings have a similar meaning. This -means a good strategy is to find the pieces of content that the most -similar embeddings to the question embeddings. - -Another possible strategy, based on the more classical keyword search, -looks like this: - -- Ask ChatGPT to extract the keywords from the question, with a prompt - like this: - -```txt -Extract keywords for a search query from the text provided. -Add synonyms for words where a more common one exists. -``` - -- Use the provided keywords to run a free-text-search and pick the top results - -Putting it in a single diagram, the two methods look like this: - - - -We have tried both on our documentation and have noticed some pros and cons. - -Letā€™s start by comparing a few results. Both are ran against the same database, -and they both use the ChatGPT `gpt-3.5-turbo` model. As there is randomness involved, -I ran each question 2-3 times and picked what looked to me like the best result. - -## Question: How do I install the Xata CLI? - -**Answer with vector search:** - - - -**Answer with keyword search:** - - - -_Verdict_: Both versions provided the correct answer, however the vector -search one is a bit more complete. They both found the correct docs page -for it, but I think our highlights-based heuristic selected a shorter -chunk of text in case of the keyword strategy. Winner: vector search. - -Score: 1-0 - -## Question: How do you use Xata with Deno? - -**Answer with vector search:** - - - -**Answer with keyword search:** - - - -**Verdict:** Disappointing result for vector search, who somehow missed the -dedicated Deno page in our docs. It did find some other Deno relevant content, -but not the page that contained the very useful example. Winner: keyword search. - -Score: 1-1 - -## Question: How can I import a CSV file with custom column types? - -**With vector search:** - - - -**With keyword search:** - - - -**Verdict:** Both have found the right page (ā€Import a CSV fileā€), but the -keyword search version managed to get a more complete answer. I did run this -multiple times to make sure itā€™s not a fluke. I think the difference comes -from how the text fragment is selected (neighbouring the keywords in case of -keyword search, from the beginning of the page in case of vector search). Winner: keyword search. - -Score: 1-2 - -## Question: How can I filter a table named Users by the email column? - -**With vector search:** - - - -**With keyword search:** - - - -_Verdict:_ The vector search did better on this one, because it found the -ā€œFilteringā€ page on which there were more examples that ChatGPT could use -to compose the answer. The keyword search answer is subtly broken, because -it uses ā€œqueryā€ instead of ā€œfilterā€ for the method name. Winner: vector search. - -Score: 2-2 - -## Question: What is Xata? - -**With vector search:** - - - -**With keyword search:** - - - -**Verdict:** This one is a draw, because both answers are quite good. The two picked different pages to -summarize in an answer, but both did a good job and I canā€™t pick a winner. - -Score: 3-3 - -## Configuration and tuning - -This is a sample Xata request used for keyword search: - -```jsonc -// POST https://workspace-id.eu-west-1.xata.sh/db/docs:main/tables/search/ask -{ - "question": "What is Xata?", - "rules": [ - "Do not answer questions about pricing or the free tier. Respond that Xata has several options available, please check https://xata.io/pricing for more information.", - "If the user asks a how-to question, provide a code snippet in the language they asked for with TypeScript as the default.", - "Only answer questions that are relating to the defined context or are general technical questions. If asked about a question outside of the context, you can respond with \"It doesn't look like I have enough information to answer that. Check the documentation or contact support.\"", - "Results should be relevant to the context provided and match what is expected for a cloud database.", - "If the question doesn't appear to be answerable from the context provided, but seems to be a question about TypeScript, Javascript, or REST APIs, you may answer from outside of the provided context.", - "If you answer with Markdown snippets, prefer the GitHub flavour.", - "Your name is DanGPT" - ], - "searchType": "keyword", - "search": { - "fuzziness": 1, - "target": [ - "slug", - { - "column": "title", - "weight": 4 - }, - "content", - "section", - { - "column": "keywords", - "weight": 4 - } - ], - "boosters": [ - { - "valueBooster": { - "column": "section", - "value": "guide", - "factor": 18 - } - } - ] - } -} -``` - -And this what we use for vector search: - -```jsonc -// POST https://workspace-id.eu-west-1.xata.sh/db/docs:main/tables/search/ask -{ - "question": "How do I get a record by id?", - "rules": [ - "Do not answer questions about pricing or the free tier. Respond that Xata has several options available, please check https://xata.io/pricing for more information.", - "If the user asks a how-to question, provide a code snippet in the language they asked for with TypeScript as the default.", - "Only answer questions that are relating to the defined context or are general technical questions. If asked about a question outside of the context, you can respond with \"It doesn't look like I have enough information to answer that. Check the documentation or contact support.\"", - "Results should be relevant to the context provided and match what is expected for a cloud database.", - "If the question doesn't appear to be answerable from the context provided, but seems to be a question about TypeScript, Javascript, or REST APIs, you may answer from outside of the provided context.", - "Your name is DanGPT" - ], - "searchType": "vector", - "vectorSearch": { - "column": "embeddings", - "contentColumn": "content", - "filter": { - "section": "guide" - } - } -} -``` - -As you can see, the keyword search version has more settings, configuring fuzziness and boosters and column weights. -The vector search only uses a filter. I would call this a plus for keyword search: you have more dials to tune the search -and therefore get better answers. But itā€™s also more work, and the results from vector search are quite good without this tuning. - -In our case, we already have tuned the keyword search for our, well, docs search functionality. -So it wasnā€™t necessarily extra work, and while playing with ChatGPT we discovered improvements -to our docs and search as well. Also, Xata just happens to have a very nice UI for tuning your -keyword search, so the work wasnā€™t hard to begin with (planning a separate blog post about that). - -There is no reason for which vector search couldnā€™t also have boosters and column weights -and the like, but we donā€™t have it yet in Xata and I donā€™t know of any other solution that -makes that as easy as we make keyword search tuning. And, in general, there is more -prior art to keyword search, but it is quite possible that vector search will catch up. - -For now, Iā€™m going to call keyword search a winner on this one. - -Score: 3-4 - -## Convenience - -Our documentation already had a search function, dog-fooding Xata, so that was quite simple -to extend to a chat bot. Xata now also supports vector search natively, but using it required adding embeddings -for all the documentation pages and figuring out a good chunking strategy. We have used the OpenAI embeddings -API for producing the text embeddings, which had a minimal cost. Winner: Keyword search - -Score 3-5 - -## Latency - -The keyword search approach needs an extra round-trip to the ChatGPT API. This adds in terms of -latency to the result started to be streamed in the UI. By my measurements, -this adds around 1.8s extra time - -**With vector search:** - - - -**With keyword search:** - - - -Note: The total and the content download times here are not relevant, because they mostly depend on how long the generated response is. Look at the ā€œWaiting for server responseā€ bar (the green one) to compare. - -Winner: Vector search - -Score: 4-5 - -## Cost - -The keyword search version needs to do an extra API call to the ChatGPT API, -on the other hand, the vector search version needs to produce embeddings -for all the documents in the database plus the question. Unless weā€™re -talking about a lot of documents, Iā€™m going to call this a tie. - -Score: 5-6 - -## Conclusion - -The score is tight! In our case we have gone with using the keyword search -for now, mostly because we have more ways of tuning it and as a result of -that it generates slightly better answers for our set of test questions. -Also, any improvements that we make to search automatically benefit both -the search and the chat use cases. As weā€™re improving our vector search -capabilities with more tuning options, we might switch to vector search, -or a hybrid approach, in the future. - -If youā€™d like to set up a similar chat bot for your own documentation, or -any kind of knowledge base, you can easily implement the above using the -Xata ask endpoint. [Create an account](https://app.xata.io/signup) for -free and join us on [Discord](https://xata.io/discord). Iā€™d be happy to -personally help you get it up and running! diff --git a/launch-week-august-2023.mdx b/launch-week-august-2023.mdx deleted file mode 100644 index 01698db9..00000000 --- a/launch-week-august-2023.mdx +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: 'Launch Week' -description: 'Join us for an exciting week filled with announcements, fun activities, and a chance to win prizes!' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/launch-week.jpg - alt: 'Launch Week' -author: Alex Francoeur -date: 08-28-2023 -published: true -tags: ['announcements', 'launch-week'] -slug: launch-week-august-2023 -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/launch-week-og.jpg ---- - -The Xata team has been hard at work all summer building the features youā€™ve been asking for. To commemorate the end of the season, we're launching our latest features in a week-long event. Join us in celebrating our launch week, complete with a content hackathon. Summer is not over yet, letā€™s enjoy one last party! šŸŽ‰ šŸŽŠ šŸ„³ šŸ¾ - -To stay up to date with the latest info, keep an eye on this page and watch for updates on [X / Twitter](https://twitter.com/xata) or [subscribe to our blog](https://xata.io/blog#subscribe-blog). Weā€™ll update our blog every day with the latest announcements. - -## Releases - -To make things more fun, weā€™re giving a riddle-like hint for each day. The first day is already revealed, but can you guess what weā€™re announcing in the other days? If yes, [tell us](https://twitter.com/xata/status/1696162078619320452?s=20) on the social network formally known as Twitter and youā€™ll get points in our [content hackathon](#xata-content-hackathon-!) (see below). - -| Day | Hint | Release blog post(s) | -| --------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Monday | The most popular data format is not necessarily the best. Or is it? | Effortlessly [import and export your data using the new CSV implementation](/blog/csv-import-export). Sounds simple, but the devil is in the details. | -| Tuesday | Snakes usually donā€™t like to be confined, but what happens if you try to put one in chains? | Our Python SDK is now [officially in general availability](/blog/announcing-the-python-sdk-ga). And if you're using Xata for an AI use case, we now have [vector and in-memory store integrations for both Python and JavaScript](/blog/langchain-integrations). | -| Wednesday | Mulder and Scully have a new case on their hands. Itā€™s a big one! | Adding files to your application just became as easy as adding a column to your database with [file attachments](/blog/file-attachments). X-Files šŸ‘½ | -| Thursday | The show is not over, a sequel is coming soon. | [SQL over HTTP](/blog/sql-over-http) is now readily available from our REST API, SDKs and web app. | -| Friday | TODO: Jason, please don't forget to add this one. | [Better support for one-to-many relationships](/blog/navigating-many-to-one), [JSON column types](/blog/json-column-type) a flurry of [getting started guides](/blog/new-getting-started-with-xata-guides) provide the developer experience you deserve. | - -## Xata content hackathon! - -To celebrate the end-of-summer launch week (Aug 28th - Sep 1st, 2023), weā€™re also organizing a content hackathon. Write a blog post, create a video, or simply spread the news and you can win: - -- šŸ’° One of 5 prizes of $500 each. -- šŸ“¦ One of the 10 special Xata swag packages (šŸ§¢ + šŸ‘• + stickers) delivered to you. - -The contest starts today and final submissions are due by September 30th, 2023 at midnight (00:00 UTC). You can win prizes by gathering points. Get points by participating in any (or multiple) of the following activities: - -- āœļø Write a blog post on your website, [dev.to](http://dev.to), or any other platform about one (or more) of the features that weā€™re announcing during launch week (10-30 points). -- šŸ“¹ Create a video and post it on YouTube or X / Twitter about one (or more) of the features that weā€™re announcing during launch week (10-30 points). -- šŸ”Š Quote-tweet any or all of our announcements during launch week on X / Twitter or create a new tweet mentioning us (10 points per tweet). -- ā¤ļø Like and repost any or all of our announcement during launch week on a social platform. (3 points for a repost, 2 points for a like). - -If you would like to participate, join our [Discord](https://xata.io/discord), follow us on [X / Twitter](https://twitter.com/xata) so you get notified of new announcements, have fun together, and help us spread the news! At the end of the hackathon, submit your entries using this [form](https://5i8caik7lja.typeform.com/to/dF0SUISW). The Xata team will assign points for each contribution. When evaluating the contributions, weā€™ll give extra points for interesting technical ideas, creativity, and reach. Some terms and conditions apply. See this [doc](https://docs.google.com/document/d/1pcRjSnXyq_RZvD9_UPH5tH5rduuEVcZw2N5Rh6MKj0Q/edit?usp=sharing) for more information. - -Let the last party of the summer begin! diff --git a/navigating-many-to-one.mdx b/navigating-many-to-one.mdx deleted file mode 100644 index 7af63564..00000000 --- a/navigating-many-to-one.mdx +++ /dev/null @@ -1,456 +0,0 @@ ---- -title: 'Reduce query round trips with improved one-to-many relations' -description: "Addressing the N+1 problem and navigating one-to-many relationships with Xata's new approach." -image: - src: 'https://raw.githubusercontent.com/xataio/mdx-blog/main/images/many-to-one-relation.jpg' - alt: 'Many to one relationship' -author: Alejandro MartĆ­nez Vieites -date: 09-01-2023 -tags: ['engineering'] -published: true -slug: navigating-many-to-one -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/many-to-one-relation-cover-og.jpg ---- - -As our [launch week](/blog/launch-week-august-2023) comes to a close, we're looking at APIs and their intricate relationships. Today, we're introducing a new syntax that simplifies navigating these relationships. While Xata already provides a great way to retrieve related data using [link columns](/docs/concepts/data-model#link) for the _many-to-one_ relationship, we're addressing the challenge of navigating the reverse [relationship](/docs/concepts/data-model#links-and-relations) (_one-to-many_). This new approach improves efficiency and helps avoid the _N + 1 problem_. - -Let's consider a basic schema. Within the schema, there's a _Posts_ table containing certain information and establishing a connection to the corresponding author. This author-related data belongs in the _Users_ table. - - - -When querying the _Posts_ table we are able to retrieve the title, likes, and the author's name (and any other fields present in the _Users_ table): - - -```ts -const records = await xata.db.posts - .select(["*", "author.*"]) - .getAll(); -``` - -```python -records = xata.data().query("posts", { - "columns": ["*","author.*"] -}) -``` - -```jsonc -// POST /db/blogs:main/tables/posts/query -{ - "columns": ["*", "author.*"] -} -``` - - - -```jsonc -// Response (xata fields have been omitted for clarity) -{ - "records": [ - { - "author": { - "id": "rec_cie05krjtojbm41fv2q0", - "name": "Tudor Golubenco" - }, - "id": "rec_cie05srjtojbm41fv2t0", - "likes": 13, - "title": "Postgres schema changes are still a PITA" - }, - { - "author": { - "id": "rec_cie05krjtojbm41fv2q0", - "name": "Tudor Golubenco" - }, - "id": "rec_cie05v3jtojbm41fv2tg", - "likes": 15, - "title": "On the performance impact of REPLICA IDENTITY FULL in Postgres" - }, - { - "author": { - "id": "rec_cie05ljjtojbm41fv2qg", - "name": "Joan Edwards" - }, - "id": "rec_cie060jjtojbm41fv2u0", - "likes": 5, - "title": "Introducing the new Xata regions: Sydney & Frankfurt" - }, - { - "author": { - "id": "rec_cie05krjtojbm41fv2q0", - "name": "Tudor Golubenco" - }, - "id": "rec_cii1kajjtoj0t82ae3ag", - "likes": 6, - "title": "Semantic or keyword search for finding ChatGPT context. Who searched it better?" - }, - { - "author": { - "id": "rec_cii1ki3jtoj0t82ae3b0", - "name": "Alex Francoeur" - }, - "id": "rec_cii1kjjjtoj0t82ae3bg", - "likes": 9, - "title": "The next era of databases are serverless, adaptive, and collaborative" - }, - { - "author": { - "id": "rec_cii1ki3jtoj0t82ae3b0", - "name": "Alex Francoeur" - }, - "id": "rec_cii1kpbjtoj0t82ae3c0", - "likes": 8, - "title": "End-to-end preview deployment workflows with Xata and Vercel" - }, - { - "author": { - "id": "rec_cii1ki3jtoj0t82ae3b0", - "name": "Alex Francoeur" - }, - "id": "rec_cii1l1rjtoj0t82ae3cg", - "likes": 20, - "title": "Modern database workflows with GitHub, Vercel, Netlify, and Xata" - } - ] -} -``` - -## The N + 1 problem - -What if we wanted to list the authors and their posts? You would need to query the _Users_ table in order to retrieve all authors: - - -```ts -const records = await xata.db.users - .select(["*"]) - .getAll(); -``` -```python -records = xata.data().query("posts", { - "columns": ["*"] -}) -``` -```jsonc -// POST /db/blogs:main/tables/users/query -{ - "columns": ["*"] -} -``` - - -```jsonc -// Response (Xata fields have been omitted for clarity) -{ - "records": [ - { - "id": "rec_cie05krjtojbm41fv2q0", - "name": "Tudor Golubenco" - }, - { - "id": "rec_cie05ljjtojbm41fv2qg", - "name": "Joan Edwards" - }, - { - "id": "rec_cii1ki3jtoj0t82ae3b0", - "name": "Alex Francoeur" - } - ] -} -``` - -And then, for each user that has been returned, make a separate request in the _Posts_ table to retrieve the posts from each author, similar to: - - -```ts -const records = await xata.db.posts - .select(["*"]) - .filter({ author: "rec_cie05krjtojbm41fv2q0"}) - .getAll(); -``` -```python -records = xata.data().query("posts", { - "columns": ["*"], - "filter": { - "author": "rec_cie05krjtojbm41fv2q0" - } -}) -``` -```jsonc -// POST /db/blogs:main/tables/posts/query -{ - "columns": ["*"], - "filter": { - "author": "rec_cie05krjtojbm41fv2q0" - } -} -``` - - -```jsonc -// Response (Xata fields have been omitted for clarity) -{ - "records": [ - { - "author": { - "id": "rec_cie05krjtojbm41fv2q0" - }, - "id": "rec_cie05srjtojbm41fv2t0", - "likes": 13, - "title": "Postgres schema changes are still a PITA" - }, - { - "author": { - "id": "rec_cie05krjtojbm41fv2q0" - }, - "id": "rec_cie05v3jtojbm41fv2tg", - "likes": 15, - "title": "On the performance impact of REPLICA IDENTITY FULL in Postgres" - }, - { - "author": { - "id": "rec_cie05krjtojbm41fv2q0" - }, - "id": "rec_cii1kajjtoj0t82ae3ag", - "likes": 6, - "title": "Semantic or keyword search for finding ChatGPT context. Who searched it better?" - } - ] -} -``` - -We are performing "1" (get the authors) + "N" (get the authors's posts) requests. This is inconvenient and also quite inefficient, since the number of requests depends on the number of user records. - -There are workarounds depending on the data set and the amount of code we are willing to write, like getting all the authors and all the posts, or using the `IN` operator and then processing the response in the app. But it would be better if our API helped us get exactly what we want. - -## Navigate links backwards - -Since the relationship is already defined by the `link` column, itā€™s a matter of providing a good API experience to get this data more easily. We have introduced [column expressions](/docs/concepts/data-model#links-and-relations) in the query endpoint so the `columns` field is no longer restricted to a list of column names. Now we are able to provide a JSON object to better define the projection we want. - -To better understand this concept, let's improve on the previous example to showcase the benefits of navigating relationships in reverse using column expressions: - - -```ts -const records = await xata.db.users - .select(["*", { - "name": "<-posts.author", - "columns": ["title"] - }]) - .getAll(); -``` -```python -records = xata.data().query("posts", { - "columns": [ - "*", - { - "name": "<-posts.author", - "columns": ["title"] - } - ] -}) -``` -```jsonc -// POST /db/blogs:main/tables/users/query -{ - "columns": [ - "*", - { - "name": "<-posts.author", - "columns": ["title"] - } - ] -} -``` - - -In this request we are asking for all the fields in the _Users_ table (`*`) plus a column expression. The expression consists of a structure with two fields: - -- `name`: When using the `<-` prefix, it signifies that we are navigating in reverse along the link defined by the `author` column in the _Posts_ table. This reverse navigation establishes a one-to-many relationship between the linked records. - -- `columns`: The `columns` field defines the projection to be applied to the table on the opposite side of this relationship. It specifies which fields from the related table should be included in the query result. - -What we get back is a list of post data nested in every author record. The default name for the nested list is `postsauthor` (i.e. table name + link name): - -```jsonc -// Response (xata fields have been omitted for clarity) -{ - "records": [ - { - "id": "rec_cie05krjtojbm41fv2q0", - "name": "Tudor Golubenco", - "postsauthor": { - "records": [ - { - "id": "rec_cie05srjtojbm41fv2t0", - "title": "Postgres schema changes are still a PITA" - }, - { - "id": "rec_cie05v3jtojbm41fv2tg", - "title": "On the performance impact of REPLICA IDENTITY FULL in Postgres" - }, - { - "id": "rec_cii1kajjtoj0t82ae3ag", - "title": "Semantic or keyword search for finding ChatGPT context. Who searched it better?" - } - ] - } - }, - { - "id": "rec_cie05ljjtojbm41fv2qg", - "name": "Joan Edwards", - "postsauthor": { - "records": [ - { - "id": "rec_cie060jjtojbm41fv2u0", - "title": "Introducing the new Xata regions: Sydney & Frankfurt" - } - ] - } - }, - { - "id": "rec_cii1ki3jtoj0t82ae3b0", - "name": "Alex Francoeur", - "postsauthor": { - "records": [ - { - "id": "rec_cii1kpbjtoj0t82ae3c0", - "title": "End-to-end preview deployment workflows with Xata and Vercel" - }, - { - "id": "rec_cii1kjjjtoj0t82ae3bg", - "title": "The next era of databases are serverless, adaptive, and collaborative" - }, - { - "id": "rec_cii1l1rjtoj0t82ae3cg", - "title": "Modern database workflows with GitHub, Vercel, Netlify, and Xata" - } - ] - } - } - ] -} -``` - -Using this syntax we are able to avoid the N + 1 problem - getting all the data we want using a single request and, therefore, avoiding any unnecessary roundtrips and repetitive code. - -More fields are available to configure the projection further: `as`, `sort`, `limit`, `offset`. The same request can be modified like the following: - - -```ts -const records = await xata.db.users - .select(["*", { - "name": "<-posts.author", - "as": "posts", - "sort": [ - { "title": "desc" } - ], - "columns": ["title"], - "limit": 1, - "offset": 1 - }]) - .getAll(); -``` -```python -records = xata.data().query("posts", { - "columns": [ - "*", - { - "name": "<-posts.author", - "as": "posts", - "sort": [ - { "title": "desc" } - ], - "columns": ["title"], - "limit": 1, - "offset": 1 - } - ] -}) -``` -```jsonc -// POST /db/blogs:main/tables/users/query -{ - "columns": [ - "*", - { - "name": "<-posts.author", - "as": "posts", // Field is returned as `posts` - "columns": ["title"], - "sort": [ // Return the nested records in this order - {"title": "desc"} - ], - "limit": 1, // Limit the amount of nested records to 1 - "offset": 1 // Skip first nested record - } - ] -} -``` - - - -```jsonc -// Response (xata fields have been omitted for clarity) -{ - "records": [ - { - "id": "rec_cie05krjtojbm41fv2q0", - "name": "Tudor Golubenco", - "posts": { - "records": [ - { - "id": "rec_cii1kajjtoj0t82ae3ag", - "title": "Semantic or keyword search for finding ChatGPT context. Who searched it better?" - } - ] - } - }, - { - "id": "rec_cie05ljjtojbm41fv2qg", - "name": "Joan Edwards", - "posts": { - "records": [ - { - "id": "rec_cie060jjtojbm41fv2u0", - "title": "Introducing the new Xata regions: Sydney & Frankfurt" - } - ] - } - }, - { - "id": "rec_cii1ki3jtoj0t82ae3b0", - "name": "Alex Francoeur", - "posts": { - "records": [ - { - "id": "rec_cii1kjjjtoj0t82ae3bg", - "title": "The next era of databases are serverless, adaptive, and collaborative" - } - ] - } - } - ] -} -``` - -For the moment, sorting options and classical pagination (limit and offset) are available for the reverse link querying. In the future, we will add support for filtering and cursor pagination. Similarly, we plan to add support for following multiple reverse links transitively. Nevertheless, for now, we know that this functionality addresses most use cases. - -## PostgreSQL implementation - -We evaluated three different strategies: - -- **Subqueries:** This approach is useful for when you need to perform operations that involve multiple tables or when you need to filter data based on the results of another query. The primary advantage is that we wouldn't have to increase the number of queries. Although, this is at the cost of introducing some complexity into the existing query and statement builder code. - -- **JOINS:** With this approach, we also maintain the same number of queries, but the code could become significantly more intricate, and we would need to deal with filtering out substantial amounts of duplicated data within each row. - -- **N+1 queries:** This approach keeps our SQL straightforward, albeit at the expense of still executing N+1 queries to the database and subsequently joining the responses together. Nonetheless, this represents an improvement because the latency (backend - PostgreSQL) is an order of magnitude lower than the latency (client - backend). - -For now, we have opted for the subquery approach due to its simplicity and its adaptability for future changes if the need arises. To further simplify the process, we are leveraging the [json_agg()](https://www.postgresql.org/docs/current/functions-aggregate.html) function to get a JSON document with all the related data ready to be returned in the response. - -Using subqueries in PostgreSQL simplifies complex queries, makes code more readable, avoids data duplication, and promotes code reusability. PostgreSQL's query optimizer can improve query performance with subqueries, which also support dynamic filtering, scalability with large datasets, and cross-platform compatibility through adherence to SQL standards. - -## Conclusion - -The new Xata querying syntax makes it possible to traverse the N:1 relationships backwards and therefore solves the N + 1 problem in a single round-trip. - -If you have feedback or questions, you can reach out to us on [Discord](https://xata.io/discord) or [X/Twitter](https://twitter.com/xata). diff --git a/new-getting-started-guides.mdx b/new-getting-started-guides.mdx deleted file mode 100644 index 0023da0e..00000000 --- a/new-getting-started-guides.mdx +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: 'New Getting Started Guides: Astro, Next.js, Nuxt, Remix, SolidStart, and SvelteKit' -description: 'Learn how to add Xata database and full-text search functionality to Astro, Next.js, Nuxt, Remix, SolidStart, and SvelteKit with our new Getting Started guides.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/new-getting-started-with-xata-guides.jpg - alt: 'Astro, Remix, SolidStart, SvelteKit, Next.js, and Nuxt logos' -author: Phil Leggetter -date: 09-01-2023 -tags: ['engineering', 'starter'] -published: true -slug: new-getting-started-with-xata-guides -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/new-getting-started-with-xata-guides-og.jpg ---- - -Our mission at Xata is to simplify the way developers work with data. This is ingrained in everything we do, from -our products to the educational material we create to enable you to be productive with Xata. So, today, as part of -[Xata launch week](/blog/launch-week-august-2023), we're excited to share six new getting started guides for these -popular full-stack JavaScript frameworks: Astro, Next.js, Nuxt, Remix, SolidStart, and SvelteKit,. - - - -The new getting started with Xata guides for [Astro][astro], [Next.js][nextjs], [Nuxt][nuxt], [Remix][remix], -[SolidStart][solidstart], and [SvelteKit][sveltekit] walk you through building a blog application and, in doing so, -help you learn some Xata fundamentals. - -The guides demonstrate features such as: - -- Using the [Xata CLI][cli] to create a database -- Defining a database schema and seeding the database by [importing a CSV][csv] -- TypeScript [code generation][codegen] with database branching -- [Querying and filtering][query-filter] -- [Full-text fuzzy search][search] - -The guides also cover some specific framework features and how those work with Xata. - -## Astro - -With the recent launch of [Astro 3.0][astro-3], now is a great time to dive into the Xata and Astro Getting Started guide. -This guide demonstrates using Xata database and full-text search in combination with Astro features -such as [Static Site Generation (SSG)][astro-ssg], [dynamic routes][astro-dynamic], and [Server Side Rendering (SSR)][astro-ssr] -via Astro "hybrid" mode. - -[Get started with Xata and Astro ā†’][astro] - -## Next.js - -Next.js is the most popular full-stack JavaScript framework on the planet. So, we've rewritten our existing guide to show how to integrate Xata -database and full-text search into a [Next.js App router][nextjs-app-router] based application with [server components][nextjs-server-components]. -The guide also covers [dynamic routes][nextjs-routes] and how to use search parameters to power full-text search. - -[Get started with Xata and Next.js ā†’][nextjs] - -## Nuxt - -The Nuxt and Xata Getting Started guide demonstrates integrating Xata database to retrieve the content via [Nuxt server/API routes][nuxt-routes], -invoked on both the server and client. The guide also covers [Vue.js conditional rendering][vue-rendering] and how to use [Nuxt dynamic parameters][nuxt-params] to drive database and full-text search queries. - -[Get started with Xata and Nuxt ā†’][nuxt] - -## Remix - -The Xata and Remix Getting Started guide shows how to build a Xata database-driven application with full-text fuzzy search. -It covers using the [`useLoaderData` Remix hook][remix-hook] to load data from a Xata database, using [Remix dynamic segments][remix-dynamic] -to pass parameters within URLs, and how to use search parameters within a Remix application to integrate Xata's full-text fuzzy search. - -[Get started with Xata and Remix ā†’][astro] - -## SolidStart - -SolidStart is currently in beta, but shows a lot of promise. This new Xata with SolidStart Getting Started guide is an opportunity to learn some -of the basics of Xata, [SolidStart][solidstart-website], and [SolidJS][solidjs]. It covers using [`createServerData$`][solid-createserverdata] and -[`useRouteData`][solid-useroutedata] to load data from a Xata database, using the [`` and ``][solid-match-switch], and [``][solid-for] -SolidJS logic operators for conditional UI rendering. It also walks through defining and accessing path parameters using [dynamic routing][solid-routing] and -how to access and use search parameters within a SolidStart application to integrate Xata's full-text fuzzy search. - -[Get started with Xata and SolidStart ā†’][solidstart] - -## SvelteKit - -The Xata and SvelteKit Getting Started guide demonstrates how to load data from a database using [SvelteKit data loading][sveltekit-data], -how to render data using [Svelte logic blocks][svelte-logic] and [page parameters][svelte-params], and how to add full-text search to -a SvelteKit application via [URL data][svelte-url-data]. - -[Get started with Xata and SvelteKit ā†’][sveltekit] - -## What should we write next? - -We'd love to hear from you if you have any feedback on the guides or if you'd like to see guides for other frameworks or technologies. -So, reach out to us on [Discord](/discord) or join us on [Twitter/X](https://twitter.com/xata). - -[astro-3]: https://astro.build/blog/astro-3/ -[astro]: /docs/getting-started/astro -[astro-ssg]: https://docs.astro.build/en/core-concepts/routing/#static-ssg-mode -[astro-dynamic]: https://docs.astro.build/en/core-concepts/routing/#dynamic-routes -[astro-ssr]: https://docs.astro.build/en/core-concepts/routing/#server-ssr-mode -[remix]: /docs/getting-started/remix -[remix-hook]: https://remix.run/docs/en/main/hooks/use-loader-data -[remix-dynamic]: https://remix.run/docs/en/main/guides/routing#dynamic-segments -[solidstart]: /docs/getting-started/solidstart -[solidstart-website]: https://start.solidjs.com/ -[solid-createserverdata]: https://start.solidjs.com/api/createServerData -[solid-useRouteData]: https://start.solidjs.com/api/useRouteData -[solid-match-switch]: https://www.solidjs.com/docs/latest/api#switchmatch -[solid-for]: https://www.solidjs.com/docs/latest/api#for -[solid-routing]: https://start.solidjs.com/core-concepts/routing -[solidjs]: https://solidjs.com/ -[sveltekit]: /docs/getting-started/sveltekit -[sveltekit-data]: https://kit.svelte.dev/docs/load -[svelte-logic]: https://svelte.dev/docs/logic-blocks -[svelte-params]: https://kit.svelte.dev/docs/routing#page-page-js -[svelte-url-data]: https://kit.svelte.dev/docs/load#using-url-data -[nextjs]: /docs/getting-started/nextjs -[nextjs-app-router]: https://nextjs.org/docs/app -[nextjs-server-components]: https://nextjs.org/docs/app/building-your-application/rendering/server-components -[nextjs-routes]: https://nextjs.org/docs/app/building-your-application/routing/dynamic-routes -[nuxt]: /docs/getting-started/nuxt -[nuxt-routes]: https://nuxt.com/docs/guide/directory-structure/server#server-routes -[nuxt-params]: https://nuxt.com/docs/guide/directory-structure/server#matching-route-parameters -[vue-rendering]: https://vuejs.org/guide/essentials/conditional.html -[cli]: /docs/getting-started/cli -[csv]: /blog/csv-import-export -[query-filter]: /docs/sdk/get -[search]: /docs/sdk/search -[codegen]: /docs/getting-started/cli#codegen diff --git a/openapi-typesafe-react-query-hooks.mdx b/openapi-typesafe-react-query-hooks.mdx deleted file mode 100644 index 51175650..00000000 --- a/openapi-typesafe-react-query-hooks.mdx +++ /dev/null @@ -1,440 +0,0 @@ ---- -title: 'From OpenAPI to type safe react-query hooks' -description: 'Learn how we build predictable API communications with the OpenAPI codegen.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/openapi-typesafe-react-query-hooks.png - alt: Xata -author: Fabien Bernard -date: 04-12-2022 -tags: ['engineering', 'ai'] -published: true -slug: openapi-typesafe-react-query-hooks ---- - -> Hey there, you are on the Xata engineering blog. Xata is a serverless data platform that makes developing on top of PostgreSQL, with TypeScript, really easy. [Sign up today](https://xata.io/)! - -The Xata engineering team values predictability at all levels of our codebase. This blog post is about how we handle API communications in a predictable way. - -## Motivation - -To us, at Xata engineering, we do not consider spending time reverse engineering APIs just to know what kind of data is available to be either fun or valuable. Instead, through [OpenAPI](https://www.openapis.org/) we get some insights into the possibilities exposed by some APIs. However, why stop there? Reading an OpenAPI spec and manually writing code isā€¦ let's say, also not our thing. - -Of course, [some](https://www.npmjs.com/package/openapi-to-ts) [libraries](https://github.com/swagger-api/swagger-js) exist to generate types from OpenAPI, but then we'd still need to manually connect them to [react-query](https://react-query.tanstack.com/) or similar fetchers, all while the generated types aren't very thorough in most cases. Even worse, in case of circular dependencies in a given OpenAPI spec, this becomes less of an option for us. - -Given these constraints, we saw potential for innovation. _We had to do something!_ And this is why we crafted [openapi-codegen](https://github.com/fabien0102/openapi-codegen). - -The goals were clear: - -1. Generate human-readable types, so we can review TypesScript files instead of a large OpenAPI YAML file. -2. Embed all possible documentation, so we can have access to the documentation directly through [IntelliSense](https://code.visualstudio.com/docs/editor/intellisense#:~:text=IntelliSense%20is%20a%20general%20term,%2C%20and%20%22code%20hinting.%22). -3. Generate ready-to-use React Hooks, with `url` and `types` baked into the component to avoid any human mistakes. -4. Be as flexible as possible - so far we can generate fetchers & react-query hooks, but let's not restrict the possibilities. - -## Let's play - -To understand some of the superpowers this generator gives you, let's create a React application that consumes the GitHub API. - -### Initialize the Project - -For simplicity, let's use [`vitejs`](https://vitejs.dev/) to bootstrap our project! - -```sh -npm create vite@latest -``` - -We'll select the framework `react` with the variant `react-ts`, this should give us a quick working environment for our application. - -Now, let's start the magic! - -```sh -npm i -D @openapi-codegen/{cli,typescript} -npx openapi-codegen init -``` - -- Select `Url` -- Paste the giphy OpenAPI url: https://api.apis.guru/v2/specs/github.com/1.1.4/openapi.yaml -- Enter `github` as namespace -- Select `react-query components` -- Enter `src/github` as folder - -Execute `npx openapi-codegen gen github` and voilĆ ! - -### Inspect what we have - -Let's take a bit of time to inspect what we have in `./src/github`. The more interesting file is `GithubComponents.ts`, this is where all our react-query components and fetchers are generated and also our entry point. We have some generated types in `Github{Parameters,Schemas,Responses}`. These are a 1-1 mapping with what we have in the OpenAPI spec. - -To finish, we have two special files: `GithubFetcher.ts` and `GithubContext.ts`. These files are generated only once and can be modified/extended to fit your needs. - -In `GithubFetcher.ts` you can: - -- inject the `baseUrl` (a default is provided) -- deal with authentication -- tweak how query strings are handled, especially if you have arrays - -and in `GithubContext.ts` you can: - -- tweak how react-query cache keys are handled (`queryKeyFn`) -- inject runtime values (hooks) to the fetcher (`fetcherOptions`) -- disable query fetching (`queryOptions`) - -### Setup react-query - -Of course, so far, we don't even have react-query installed, let's do this and setup our application. - -```sh -npm i react-query -``` - -Add the `queryClient` - -```tsx title="app.tsx" -import { QueryClient, QueryClientProvider } from \"react-query\"; - -const queryClient = new QueryClient(); - -function App() { - return ( - - - - ); -} - -function Users() { - return
todo
; -} - -export default App; -``` - -We can run `yarn dev` and see a white page with "todo". - -### Start fetching - -Let's try to fetch some users! - -```tsx -import { QueryClient, QueryClientProvider } from \"react-query\"; -import { useSearchUsers } from \"./github/githubComponents\"; - -const queryClient = new QueryClient(); - -function App() { - return ( - - - - ); -} - -function Users() { - const { data, error, isLoading } = useSearchUsers({ - queryParams: { q: \"fabien\" }, - }); - - if (error) { - return ( -
- ); - } - - if (isLoading) { - return
Loadingā€¦
; - } - - return ( -
    - {data?.items.map((item) => ( -
  • {item.login}
    • - ))} -
- ); -} - -export default App; -``` - -And voilĆ ! Without knowing the API, just playing with the autocompletion, I'm able to fetch a list a users! šŸ„³ The types even give us a hint that `error.documentation_url` was a thing, quite cool, but let's see if this is actually working! - -### Error management - -Let's refresh the page many times, until we reach the API rate limit šŸ˜… Eventually, we don't see the error message, nor the documentation link. - -This is where `GithubFetcher.ts` needs to be tweaked! Errors are indeed a bit trickier, as an error can be proper object back from the API or anything else (text response instead of JSON, or something less predictable if the network is down), so we need to make sure we received a known format in our application. - -```diff -// GithubFetcher.ts -@@ -1,4 +1,5 @@ -+import { BasicError } from \"./githubSchemas\"; - -@@ -43,8 +43,18 @@ export async function githubFetch< - } - ); - if (!response.ok) { -- // TODO validate & parse the error to fit the generated error types -- throw new Error(\"Network response was not ok\"); -+ let payload: BasicError; -+ try { -+ payload = await response.json(); -+ } catch { -+ throw new Error(\"Network response was not ok\"); -+ } -+ -+ if (typeof payload === \"object\") { -+ throw payload; -+ } else { -+ throw new Error(\"Network response was not ok\"); -+ } - } - return await response.json(); - } -``` - -Here we need to make sure to validate the error format, since everything is optional in `BasicError` GitHub API and `BasicError` have a `message: string` property, I can safely throw an `Error`. And just like this, we have nice and type-safe error handling. - -### Adding some states - -Of course, [React is all about state](https://twitter.com/TejasKumar_/status/1513123801994866689) isn't it? Soā€¦ let's try! - -```tsx -function Users() { - const [query, setQuery] = useState(\"\"); - const { data, error, isLoading } = useSearchUsers( - { - queryParams: { q: query }, - }, - { - enabled: Boolean(query), - } - ); - - if (error) { - return ( -
-

{error.message}

- Documentation -
- ); - } - - return ( -
- setQuery(e.target.value)} /> - {isLoading ? ( -
Loadingā€¦
- ) : ( -
    - {data?.items.map((item) => ( -
  • {item.login}
    • - ))} -
- )} -
- ); -} -``` - -Now I can search for users, and also reach the API rate limit way quicker! šŸ˜… I guess this is a good time to introduce authentication handling in our little application. - -### Authentication - -To keep it simple for our demo purposes, let's take a GitHub developer token and store it in localStorage. **This is unsafe and you probably shouldn't do this in production**. If the token is not set, ask for it, if it's there, show the data and provide a disconnect button. We will need to access this `token` at runtime later, with this in mind, let's create a `Auth.tsx` file, that isolates our authentication logic. - -```tsx title="auth.tsx" -import React, { createContext, useContext, useState } from \"react\"; -import { useQueryClient } from \"react-query\"; - -const authContext = createContext<{ token: null | string }>({ - token: null, -}); - -export function AuthProvider({ children }: { children: React.ReactNode }) { - const key = \"githubToken\"; - const [token, setToken] = useState(localStorage.getItem(key)); - const [draftToken, setDraftToken] = useState(\"\"); - const queryClient = useQueryClient(); - - return ( - - {token ? ( - <> - {children} - - - ) : ( -
-
{ - e.preventDefault(); - setToken(draftToken); - localStorage.setItem(key, draftToken); - }} - > -

Please enter a personal access token

- setDraftToken(e.target.value)} - > -
-
- )} - - ); -} - -export const useToken = () => { - const { token } = useContext(authContext); - - return token; -}; -``` - -Now, we can wrap our App with the AuthProvider, so the `token` is available in the entire application: - -```tsx title="app.tsx" -function App() { - return ( - - - - - - ); -} -``` - -and inject the `token` in `GithubContext.ts` - -```diff title="githubContext.ts" ---- a/src/github/githubContext.ts -+++ b/src/github/githubContext.ts -@@ -1,4 +1,6 @@ - import type { QueryKey, UseQueryOptions } from \"react-query\"; -+import { useToken } from \"../useAuth\"; - import { QueryOperation } from \"./githubComponents\"; - - export type GithubContext = { -@@ -6,7 +8,9 @@ export type GithubContext = { - /** - * Headers to inject in the fetcher - */ -- headers?: {}; -+ headers?: { -+ authorization?: string; -+ }; - /** - * Query params to inject in the fetcher - */ -@@ -36,14 +40,22 @@ export function useGithubContext< - TData = TQueryFnData, - TQueryKey extends QueryKey = QueryKey - >( -- _queryOptions?: Omit< -+ queryOptions?: Omit< - UseQueryOptions, - \"queryKey\" | \"queryFn\" - > - ): GithubContext { -+ const token = useToken(); -+ - return { -- fetcherOptions: {}, -- queryOptions: {}, -+ fetcherOptions: { -+ headers: { -+ authorization: token ? `Bearer ${token}` : undefined, -+ }, -+ }, -+ queryOptions: { -+ enabled: token !== null && (queryOptions?.enabled ?? true), -+ }, - queryKeyFn: (operation) => { - const queryKey: unknown[] = hasPathParams(operation) - ? operation.path -``` - -And that's it! Now we have everything in place to create an amazing application around github API. - -How is this working? Let's have a look at the generated `useSearchUsers`: - -```ts -export const useSearchUsers = ( - variables: SearchUsersVariables, - options?: Omit< - reactQuery.UseQueryOptions< - SearchUsersResponse, - | Responses.NotModified - | Responses.ValidationFailed - | Responses.ServiceUnavailable, - SearchUsersResponse - >, - \"queryKey\" | \"queryFn\" - > -) => { - // 1. we retrieve our custom auth logic - const { fetcherOptions, queryOptions, queryKeyFn } = - useGithubContext(options); - return reactQuery.useQuery< - SearchUsersResponse, - | Responses.NotModified - | Responses.ValidationFailed - | Responses.ServiceUnavailable, - SearchUsersResponse - >( - queryKeyFn({ - path: \"/search/users\", - operationId: \"searchUsers\", - variables, - }), - // 2. the custom `headers.authorization`, part of `fetcherOptions` is passed to the fetcher - () => fetchSearchUsers({ ...fetcherOptions, ...variables }), - { - ...options, - ...queryOptions, - } - ); -}; -``` - -## What about query caching? - -One of the key features of react-query, is to have a query cache. By default, openapi-codegen provides you with a key cache that fits the URL scheme. - -For example: - -`useSearchUsers( { queryParams: { q: \"fabien\" } }` will produce the following cache key: `[\"search\", \"users\", { q: \"fabien\" }]`. - -So if I want to invalidate the users list, I can call: - -```tsx -import { useQueryClient } from \"react-query\"; - -const MyComp = () => { - const queryClient = useQueryClient(); - - const onAddUser = () => { - queryClient.invalidateQueries([\"search\", \"users\"]); - }; - - /* ... */ -}; -``` - -Note: This is just a default behavior, if you have more specific needs, you can always tweak `GithubContext#queryKeyFn`. - -## Help & feedback - -This project is still relatively early stage (but it strictly follows semver, so if we break the API, you will know it!). We're always happy to have feedback and help you if you have any questions. If you'd like to browse the code or take it for a spin, be sure to check out the [repo on GitHub](https://github.com/fabien0102/openapi-codegen). - -Happy coding folks! diff --git a/rag-vector-sveltekit-xata.mdx b/rag-vector-sveltekit-xata.mdx deleted file mode 100644 index fae83f1a..00000000 --- a/rag-vector-sveltekit-xata.mdx +++ /dev/null @@ -1,420 +0,0 @@ ---- -title: 'Building a Retrieval-Augmented Generation Chatbot with SvelteKit and Xata Vector Search' -description: 'Learn how to create a Retrieval-Augmented Generation Chatbot using Xata, SvelteKit, LiteLLM, TailwindCSS and Vercel.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/rag-sveltekit-xata/rag-sveltekit-xata-illustration.jpeg - alt: Chatbot with Xata -author: Rishi Raj Jain -date: 02-28-2024 -tags: ['fpGenerativeAi'] -published: true -slug: rag-vector-sveltekit-xata -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/rag-sveltekit-xata/rag-sveltekit-xata-illustration.jpeg ---- - -In this post, you'll create a Retrieval-Augmented Generation Chatbot using Xata, SvelteKit, LiteLLM, TailwindCSS and Vercel. You'll learn how to: - -- Set up Xata -- Create a schema with different column types -- Using LiteLLM to Create Vector Embeddings -- Using Xata to Store Vector Embeddings with Metadata -- Using Xata Vector Search to Create Relevant Context -- Using Replicate Predictions to Prompt LLAMA 2 70B Chat Model -- Wiring Up Chatbot UI in SvelteKit - -## Before you begin - -### Prerequisites - -You'll need the following: - -- A [Xata](https://xata.io/) account -- [Node.js 18](https://nodejs.org/en/blog/announcements/v18-release-announce) or later -- An [OpenAI](https://platform.openai.com) account -- A [Replicate](https://replicate.com) account -- A [Vercel](https://vercel.com) Account - -### Tech Stack - -| Technology | Description | -| --------------------------------------------- | ------------------------------------------------------------------------------------ | -| [Xata](https://xata.io) | Serverless database platform for scalable, real-time applications | -| [SvelteKit](https://kit.svelte.dev) | UI framework that uses a compiler to let you write breathtakingly concise components | -| [LiteLLM](https://github.com/BerriAI/litellm) | Call all LLM APIs using the OpenAI format | -| [Replicate](https://replicate.com) | Platform to run and fine-tune open-source models | -| [TailwindCSS](https://tailwindcss.com) | CSS framework for building custom designs | -| [Vercel](https://vercel.com) | A cloud platform for deploying and scaling web applications | - -## Setting up a Xata Database - -After you've created a Xata account and are logged in, create a database. - -![Create Xata database](/images/rag-sveltekit-xata/rag-sveltekit-xata-01.png) - -The next step is to create a table, in this instance `vectors`, that will contain all the vector embeddings along with their metadata. - -![Create vectors table](/images/rag-sveltekit-xata/rag-sveltekit-xata-02.png) - -Letā€™s move on to adding relevant columns in the table you've just created. - -## Creating the Schema - -In the `vectors` table, you want to store all the vector embeddings along with metadata (such as string content) so that you can use metadata of the relevant vector(s) to create context for prompting the LLAMA 2 70B Chat model. - -First, click **+ Add column** and select `String`. - -![Create string type column](/images/rag-sveltekit-xata/rag-sveltekit-xata-03.png) - -Proceed with adding the column name as `contents`. This column is responsible for storing the metadata as `String` type. - -![Name the column as contents](/images/rag-sveltekit-xata/rag-sveltekit-xata-04.png) - -Next, proceed to add another column name as `embedding`. This column is responsible for storing the vector embedding generated via OpenAI as `vector` type. - -Among multiple [embedding models offered by OpenAI](https://platform.openai.com/docs/guides/embeddings), the `text-embedding-3-small` model is one of the newest and performant embedding models. The length of the vector embedding generated via the `text-embedding-3-small` model is 1536. - -![Create vector type column](/images/rag-sveltekit-xata/rag-sveltekit-xata-05.png) - -Enter the default value of the length of the vector embedding equal to 1536. - -![Name the column as embedding and set the default embedding vector length](/images/rag-sveltekit-xata/rag-sveltekit-xata-06.png) - -Lovely! With all that done, the final schema shall be shown as below šŸ‘‡šŸ» - -![View the database schema](/images/rag-sveltekit-xata/rag-sveltekit-xata-07.png) - -## Setting up the project - -Clone the app repository and follow this tutorial; you can fork the project by running the following command: - -```bash -git clone https://github.com/rishi-raj-jain/xata-rag-chatbot -cd xata-rag-chatbot -pnpm install -``` - -## Configure Xata with SvelteKit - -To seamlessly use Xata with SvelteKit, install the Xata CLI globally: - -```bash -npm install -g @xata.io/cli -``` - -Then, authorize the Xata CLI so it is associated with the logged in account: - -```bash -xata auth login -``` - -![Create xata cli api key](/images/rag-sveltekit-xata/rag-sveltekit-xata-08.png) - -Great! Now, initialize your project locally with the Xata CLI command: - -```bash -xata init --db https://Rishi-Raj-Jain-s-workspace-80514q.us-east-1.xata.sh/db/rag-chatbot -``` - -Use the following answers to the Xata CLI one-time setup question prompts to integrate Xata with SvelteKit: - -- `Yes` when prompted to add `.env` to `.gitignore`. -- `TypeScript` when prompted to select the language to generate code and types from your Xata database. -- `src/xata.server.ts` when prompted to enter the output path for the generated code by the Xata CLI. - -![Set up xata with sveltekit](/images/rag-sveltekit-xata/rag-sveltekit-xata-09.png) - -## Using LiteLLM to Create Vector Embeddings - -To use OpenAI Embedding models using LiteLLM, we need to make sure that the `OPENAI_API_KEY` exists as an environment variable. Refer to the following OpenAI doc on how to find your API Key: [Where do I find my OpenAI API Key?](https://help.openai.com/en/articles/4936850-where-do-i-find-my-openai-api-key) - -```bash -# OpenAI API Key -OPENAI_API_KEY="sk-..." -``` - -With LiteLLM, you can call 100+ LLMs with the same OpenAI-like input and output. To create an embedding vector from a string content, we use the `embedding` method from `litellm` with `text-embedding-3-small` as the model. By default, the length of the embedding vector will be `1536` for text-embedding-3-small. Using the OpenAI output format, we obtain the embedding vector from the first object in the `data` array returned. - -```typescript -// File: src/routes/api/chat/+server.ts - -import { embedding } from 'litellm'; - -export async function POST({ request }: RequestEvent) { - // Generate embeddings of a message using OpenAI via LiteLLM - const embeddingData = await embedding({ - model: 'text-embedding-3-small', - input: 'Rishi is enjoying using LiteLLM' - }); - - // Using the OpenAI output format, obtain the embedding vector stored in - // the first object of the data array - const getEmbeddingVector = embeddingData.data[0].embedding; -} -``` - -## Using Xata to Store Vector Embeddings with Metadata - -Weā€™ll use the Xata Serverless database platform (powered by PostgreSQL) to store the content from which the embedding vector was created, and the embedding vector itself. Itā€™s as easy as inserting a record in your database. Hereā€™s the code on how to add each vector with itā€™s content to the `vectors` table in your Xata database. - -```typescript -// Upsert the vector with description to be used further -// as the context to upcoming questions -xata.db.vectors.create({ - contents: message, - embedding: embeddingData.data[0].embedding, -}), -``` - -Storing vectors in a database helps us create context when generating chatbot responses on the fly including the latest content added. To add embedding vectors along with metadata dynamically, we create a [Server Side Rendered SvelteKit endpoint](https://kit.svelte.dev/docs/routing#server) to accept `messages` (a string array) in the POST request. Using messages, we create embeddings and insert them into our Xata database. - -```typescript -// File: src/routes/api/context/warm/+server.ts - -import { embedding } from 'litellm'; -import { getXataClient } from '@/xata.server'; - -// Load the in-memory Xata client -const xata = getXataClient(); - -export async function POST({ request }: RequestEvent) { - // Set of messages to create vector embeddings on - const { messages = [] } = await request.json(); - - // Call the OpenAI API to get embeddings on the messages - const generatedEmbeddings = await Promise.all( - messages.map((input: string) => embedding({ input, model: 'text-embedding-3-small' })) - ); - - // Insert all of them into Xata Embedding with the content - await Promise.all( - generatedEmbeddings.map((embeddingData, index) => - // Upsert the vector with description to be further as the context to upcoming questions - xata.db.vectors.create({ - contents: messages[index], - embedding: embeddingData.data[0].embedding - }) - ) - ); -} -``` - -## Using Xata Vector Search to Create Relevant Context - -Weā€™ll use the Xata Vector Search to perform similarity search on your data based on the latest queryā€™s dynamically generated embedding vector. Itā€™s as easy as retrieving a record in your database. Hereā€™s the code on how to find the 5 most similar embedding vectors to the latest user query vector in your Xata database. - -```typescript -// Fetch the relevant set of records based on the embedding -const relevantRecords = await xata.db.vectors.vectorSearch( - // Column Name - 'embedding', - // Embedding Vector - embeddingData.data[0].embedding, - // Count of relevant vector embeddings - { size: 5 } -); -``` - -With each relevant embedding vector, each record of your Xata database is returned. This allows us to obtain the metadata associated with these vectors and use them to create the context for the AI to answer with. In the code below, we simply create a string containing all the metadata from the relevant vectors returned in our search. - -```typescript -// File: src/routes/api/context/warm/+server.ts - -import { embedding } from 'litellm'; -import { getXataClient } from '@/xata.server'; - -// Load the in-memory Xata client -const xata = getXataClient(); - -export async function POST({ request }: RequestEvent) { - // Set of messages to create vector embeddings on - const { messages = [] } = await request.json(); - - // Get the latest question stored in the last message of the chat array - const userMessages = messages.filter((i: any) => i.role === 'user'); - const input = userMessages[userMessages.length - 1].content; - - // Generate embeddings of the latest question using OpenAI - const embeddingData = await embedding({ input, model: 'text-embedding-3-small' }); - - // Fetch the relevant set of records based on the embedding - const relevantRecords = await xata.db.vectors.vectorSearch('embedding', embeddingData.data[0].embedding, { size: 5 }); - - // Combine all the metadata of the relevant vectors - const systemContext = relevantRecords.records.map((i) => i.contents).join('\n'); -} -``` - -## Using Replicate Predictions to Prompt LLAMA 2 70B Chat Model - -To prompt Metaā€™s LLAMA 2 70B Chat model, weā€™ll use Replicate which allows us to run the model with just an API call. In our case, we make use of the `replicate` package, saving us from the need to write a fetch api by ourselves and handle streaming responses natively. We need to make sure that the `REPLICATE_API_TOKEN` is set as an environment variable. Go to your [Replicateā€™s Account Settings > API tokens](https://replicate.com/account/api-tokens) to obtain your Replicate API token. - -```typescript -import Replicate from 'replicate' - -// Instantiate the Replicate API -const replicate = new Replicate({ - auth: process.env.REPLICATE_API_TOKEN, -} -``` - -Using Vercelā€™s AI SDK baked-in `experimental_buildLlama2Prompt` function, weā€™re able to: - -1. Pass instructions so that the code blocks in the markdown-like responses generated by AI are prepended with \`\`\`. This'll help us highlight them in our frontend setup via a syntax highlighter. This is done via creating a message with the role of `assistant`and passing our relevant instructions in the`content` field. -2. Pass the dynamically generated context (as string) for the AI to consider before generating responses. This is done via creating a message with the role of `system` and passing our relevant context string in the `content` field. -3. Pass the whole conversation between user and chatbot as `messages`, which contains each message with its relevant `content` and `role` field. - -````typescript -import { experimental_buildLlama2Prompt } from 'ai/prompts'; - -// Now use Replicate LLAMA 70B streaming to perform the autocompletion with context -const response = await replicate.predictions.create({ - // You must enable streaming. - stream: true, - // The model must support streaming. See https://replicate.com/docs/streaming - model: 'meta/llama-2-70b-chat', - // Format the message list into the format expected by Llama 2 - // @see https://github.com/vercel/ai/blob/99cf16edf0a09405d15d3867f997c96a8da869c6/packages/core/prompts/huggingface.ts#L53C1-L78C2 - input: { - prompt: experimental_buildLlama2Prompt([ - { - // create a system content message to be added as - // the llama2prompt generator will supply it as the context with the API - role: 'system', - content: systemContext - }, - { - // create a system instruction - // make sure to wrap code blocks with ``` so that the svelte markdown picks it up correctly - role: 'assistant', - content: `When creating responses sure to wrap any code blocks that you output as code blocks and not text so that they can be rendered beautifully.` - }, - // also, pass the whole conversation! - ...messages - ]) - } -}); -```` - -## Wiring Up Chatbot UI in SvelteKit - -Using Vercel's AI SDK weā€™re able to save tremendous time in setting up a reactive chatbot UI in SvelteKit. The `useChat` utility from `ai/svelte` encapsulates the frontend logic to maintaining the array of messages exchanged between the user and the chatbot, submitting form responses to the relevant API endpoint with the messages exchanged, and stream content right into the UI from the streaming API response. - -```html - - -
-
-
- - - -
- {#each $messages as message} -
- {capitalizeFirstLetter(message.role)} -
- -
- {/each} -
-
-
-``` - -Using the `svelte-markdown` package, weā€™re able to visually format the responses returned by the chatbot API route for the user. Breaking HTML into components of their own tags (such as `` and ``), weā€™re able to selectively pass our own Svelte components containing their own processing logic. In this example, we create our custom `Code` svelte component to touch up the `` elementā€™s visual appearance. - -```html - - - -``` - -Using `svelte-highlight` package, weā€™ll specifically highlight the code blocks in the chatbot responses to make them more visually appealing. Within just a few lines, weā€™re able to create HTML classes for each code block specifically based on its language (by default, weā€™ve set it to `typescript`). - -```html - - - -``` - -To make sure the specific HTML classes have their respective CSS styles present, we use the `GitHub Light Theme` (exported by `svelte-highlight`) in the appā€™s parent [layout file](https://kit.svelte.dev/docs/advanced-routing#advanced-layouts-layout). - -```html - - - - {@html github} - - - -``` - -Here's a preview of what we've successfully created with SvelteKit and Xata šŸ‘‡šŸ» - -![Conversational UI with Xata](/images/rag-sveltekit-xata/rag-sveltekit-xata-10.png) - -## Deploy to Vercel - -The repository, is now ready to deploy to Vercel. Use the following steps to deploy: - -- Start by creating a GitHub repository containing your app's code. -- Then, navigate to the Vercel Dashboard and create a **New Project**. -- Link the new project to the GitHub repository you just created. -- In **Settings**, update the **Environment Variables** to match those in your local `.env` file. -- Deploy! šŸš€ - -## More Information - -For more detailed insights, explore the references cited in this post. - -| Resource | Link | -| --------------------- | -------------------------------------------------------- | -| GitHub Repo | https://github.com/rishi-raj-jain/xata-rag-chatbot | -| SvelteKit with Xata | https://xata.io/docs/getting-started/sveltekit | -| Xata Vector Search | https://xata.io/docs/sdk/vector-search | -| How to Prompt Llama 2 | https://huggingface.co/blog/llama2#how-to-prompt-llama-2 | -| SvelteKit AI SDK | https://sdk.vercel.ai/docs/guides/frameworks/sveltekit | - -## Whatā€™s next? - -We'd love to hear from you if you have any feedback on this tutorial, would like to know more about Xata, or if you'd like to contribute a community blog or tutorial. Reach out to us on [Discord](https://discord.com/invite/kvAcQKh7vm) or join us on [X | Twitter](https://twitter.com/xata). Happy building šŸ¦‹ diff --git a/searching-for-retro-games.mdx b/searching-for-retro-games.mdx deleted file mode 100644 index 1c8605c8..00000000 --- a/searching-for-retro-games.mdx +++ /dev/null @@ -1,494 +0,0 @@ ---- -title: 'Searching for retro games with Xata & Next.js 13' -description: 'Creating a full-stack web app to search retro games data and prioritize highly-rated games in the results.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/search-retro-games.png - alt: Data platforms -author: Anjana Vakil -date: 11-02-2022 -tags: ['nextjs', 'search'] -published: true -slug: search-retro-games ---- - -_Long ago, in the beautiful kingdom of JavaScript surrounded by cascades and DOM trees..._ -_legends told of an omnipotent and omniscient Gaming Power that resided in a hidden app._ - -_It's hidden because you haven't built it yet._ -_And the time of destiny for Princess ~~Zelda~~ You is drawing near._ - -## An epic quest - -In this tutorial we'll build a full-stack web app that allows us to search a collection of retro games data to find the old-school console games of our dreams. And I hope you like adventure - for this will be a treacherous journey through the bleeding edge of recently released web technologies! - -We'll learn how to: - -- use [Xata](https://xata.io)'s serverless platform to store & retrieve data without needing a database -- build an app with React Server & Client Components using [Next.js 13](https://nextjs.org/blog/next-13) and its `app/` directory -- implement full-text search with boosting via the Xata SDK - -Before we begin, make sure you have [Node/npm](https://nodejs.org/en/) installed, and download the [games.csv](https://github.com/vakila/search-xata-next13/raw/main/util/games.csv) data we'll be working with. You can optionally check out the [completed tutorial code](https://github.com/vakila/search-xata-next13) for reference. - -The games may be old, but the tech is brand spankin' new - so let's play! - -## Backstory - -My friend & avid retro gamer [Sara Vieira](https://github.com/saravieira) had compiled a great collection of retro games data for her awesome site [letsplayretro.games](https://letsplayretro.games), a Next [app](https://github.com/SaraVieira/lets-play-retro-games) which already had a ton of great features. But one feature Sara still wished for was full-text search over not just games' names, but all their metadata, with the ability to prioritize (aka "boost") the highest-rated games in the search results. - -Since every minute spent coding is one less minute spent playing NES, neither of us wanted to waste a lot of time on making this happen. That's when it dawned on me that I knew of a way to easily implement full-text, boosted search over her tabular data: [Xata](https://xata.io) provides it out of the box! - -## It's-a me, Xata! - -Xata is a serverless data platform that's new on the Jamstack scene, and aims to make life easier for developers to build full-stack apps need without worrying about the details of a scalable, performant backend database. Let's give it a whirl. - -### Set up Xata CLI - -Sign up for a free account at [xata.io](https://xata.io) to get started. - -Then, from the command line install the [CLI](https://xata.io/docs/getting-started/cli) and authenticate with your Xata account: - -```bash -npm i --location=global @xata.io/cli -xata auth login -``` - -### Import CSV data - -Now we need to get our games data into Xata. Sara gave us a file `games.csv` with information about 7.3K games, and conveniently Xata offers a CSV importer that lets us create a database table from that file, with an automatically-inferred schema. - -Import the file with the following command. Xata will ask you to set up a new workspace & database (I called my database `retrogames` but you can name yours as you please), then create a new table in that database named `games`: - -```bash -xata import csv games.csv --table=games -``` - -Now let's take a look at our data in Xata's web interface. - -### Browse the dashboard - -Open your newly-created database in your browser with the command: - -```bash -xata browse -``` - -This brings us to a spreadsheet-like view of our games data in the Xata dashboard. We can use the dashboard to browse, query, edit, and search our data, view and edit our databases, tables and schemas, and more. - - - -For our app's purposes, these columns of the data are going to be particularly useful: - -- `id`: the game's unique ID in the database -- `name`: the game's name -- `totalRating`: the game's average rating on [IGDB](https://igdb.com) -- `cover`: URL of the game's cover image - -### Search all the things! - -The "Search" tab lets us try out Xata's built-in full-text search capabilities. Whatever search term we enter will be matched against any of the text-based columns in our database; each result is assigned a relevance score, with most relevant results appearing first. - - - -Using Xata's "Boost" feature, we can increase the relevance score of certain results by telling Xata which information we care most about. For example, we can add a numeric booster based on `totalRating` to have the highest-rated games appear higher in the search results. - - - -To deliver this awesome, customized search experience to our users, we'll use Xata's SDK to interact with our database from our app. On the Search page in the dashboard, click "Get code snippet" to reveal the SDK code for performing that search, including any boosters applied: - -```tsx -// Generated with CLI -import { getXataClient } from './xata'; -const xata = getXataClient(); - -const { records } = await xata.search.all('demon', { - tables: [ - { - table: 'games', - boosters: [{ numericBooster: { column: 'totalRating', factor: 2 } }] - } - ], - fuzziness: 0, - prefix: 'phrase' -}); - -console.log(records); -``` - -We'll use this code later to search our Xata data from our web app. But that app doesn't exist yet, so first we'll need to create it! - -## Next.js 13, fresh on the scene - -Because life is full of contradictions, we're going to serve up our super vintage games data with the newest, most cutting-edge fullstack web framework we know: [Next.js 13](https://nextjs.org/blog/next-13), which is only about a week old at the time of writing. - -Next.js is beloved by React developers for its server-side rendering capabilities, which make apps more performant by letting us pre-load data and render React components on the server when we can, rather sending all that JS over the wire and making the client to do the heavy lifting on each request. - -With v13 and its experimental `app/` directory, Next is now changing up the way we think about architecting & fetching data in Next apps, prioritizing server-side rendering as default via React [Server Components](https://beta.nextjs.org/docs/rendering/server-and-client-components), while still allowing us to write hook-based Client Components when we need them. - -The Next and React features this new architecture is based on are still experimental and may change, but it's worth wrapping our heads around the new direction in which the React/Next community is headed - so let's dive in and try them out! - -### Bootstrapping Next with `app/` directory - -The handy tool `create-next-app` helps us bootstrap a new Next app, and in the latest version we can try out the experimental new `app/` directory. - -Back on the command line, create an app with the following command, walking through the prompts to name the project as you please and use TypeScript (or JavaScript if you prefer): - -```bash -npx create-next-app@latest --experimental-app -``` - -Exploring our newly-created directory, we notice the following structure: - -```bash -app/ - layout.tsx (or .jsx if using JS) - page.tsx -pages/ - api/ - hello.ts (or .js) -``` - -This represents a partial [migration](https://beta.nextjs.org/docs/upgrade-guide#migrating-from-pages-to-app) to the new `app/` directory architecture, in which `app/page.tsx` will be rendered at our app's `/` route, using the default layout `app/layout.tsx`. However, the new `app/` directory doesn't yet support API routes, so we're still using the old structure `pages/api/hello.ts` to create the [API route](https://nextjs.org/docs/api-routes/introduction) `/api/hello`. See the [Next.js 13](https://beta.nextjs.org/docs/routing/fundamentals#) documentation for more details. - -To build our search app, we'll need to write all three of these: - -- Server Components -- Client Components -- API routes - -In order to fetch the data we'll need for these, we first need to set up the Xata SDK so that our app can talk to the database we created earlier in Xata. - -### Connect to Xata - -In the new directory created by `create-next-app`, [initialize a Xata project](https://xata.io/docs/getting-started/cli#project-mode) with the command: - -```bash -xata init -``` - -In the prompts that follow, select the database you created earlier, and choose to generate TypeScript code (or JS code with ESM if you chose not to use TS) for the Xata client. This will create a `xata.ts` file at the path you specify; you can use the default path `src/xata.ts` or change it to e.g. `util/xata.ts` as I did. - -After running the init script, you'll have a `.env` file with the environment variables the Xata client needs to run, and a `xata.ts` file which exports a `getXataClient()` function we can now use to create a client object that can read our database on Xata. - -### Create a Server Component - -By default, components in the `app/` directory (like `app/page.jsx`) are [Server Components](https://beta.nextjs.org/docs/rendering/server-and-client-components#server-components). Server components are great for parts of our app that don't require client-side interactivity. - -Let's try writing our first server component to load & display data about a certain game. We'll make the route for this page `/games/[gameID]`, where `[gameID]` is a [dynamic route segment](https://beta.nextjs.org/docs/routing/defining-routes#dynamic-segments) matching the ID of a game in our dataset, e.g. `/games/123` will show info for "Clash at Demonhead". - - - -To make the new route, create a new file at the path `app/games/[gameID]/page.tsx`. Now, we need to flesh out that file to load data from Xata into a Server Component. - -In the new `page.tsx` file, import & call `getXataClient()` to instantiate a new Xata client: - -```tsx -// app/games/[gameID]/page.tsx - -import { getXataClient } from '../../../util/xata'; // or the path you chose during 'xata init' - -const xata = getXataClient(); -``` - -Now, we can query our data with the Xata SDK by calling methods on `xata.db[tableName]`, in our case `xata.db.games`. - -To retrieve a game by its ID, for example, we can use [`.read()`](https://xata.io/docs/sdk/get): - -```tsx -const clashAtDemonhead = await xata.db.games.read('123'); -``` - -Unlike with previous versions of Next, when fetching data in `app/` Server Components we don't need anything fancy tricks like `getServerSideProps`. We can `await` our data-fetching functions like we would anywhere else! - -In `page.tsx`, export a `Page()` function that captures the ID from our dynamic `gameID` route segment from the `params` object passed in by Next, and passes that ID to the call to `.read()`. We can then use properties of the `game` record, e.g. its `name`, `cover` image, `summary` text and `console`, to render a basic page: - -```tsx -// app/games/[gameID]/page.tsx - -import { getXataClient } from '../../../util/xata'; // or the path you chose during 'xata init' - -const xata = getXataClient(); - -export default async function Page({ params }: { params: { gameID: string } }) { - const game = await xata.db.games.read(params.gameID); - - if (!game) return

Game not found

; - - return ( -
-

{game.name}

- {game.cover && } -

{game.summary}

-

{game.console}

-
- ); -} -``` - -To see the new page in action, start up a local Next server with the command: - -```bash -npm run dev -``` - -Then navigate to `localhost:3000/games/123` and behold your beautiful new server-rendered component, which automatically uses the default layout in `app/layout.tsx`. - -Server Components are great for pages that don't require interactivity, like our game-info page, because they can be more performantly rendered server-side. But what we really wanted in this app was a search page that dynamically loads data based on user input; for that, we're going to need a Client Component. - -### Create a Client Component - -[Client Components](https://beta.nextjs.org/docs/rendering/server-and-client-components#client-components) are what we might think of as "traditional" React components that allow us to power client-side interactions with hooks like `useState()`. Now that the `app/` directory makes Server Components the default, we have to explicitly designate our Client Components with the `'use client';` directive. - -Create a new `Search` Client Component by creating a new file `app/search.tsx` with `'use client';` on the first line and exporting a function called `Search()` that renders a basic search input: - -```tsx -// app/search.tsx -'use client'; - -export default function Search() { - return ( -
- -
- ); -} -``` - -We want this component to show up on our app's home page at the `/` route, so in `app/page.tsx` import the `Search` component and use it to replace the default content in the `Home` component generated by `create-next-app`. While you're at it, why not change the page heading to something more fun? - -```tsx -// app/page.tsx -import styles from './page.module.css'; -import Search from './search'; - -export default function Home() { - return ( -
-
-

Search Retro Games!

- - -
-
- ); -} -``` - -Navigate to `localhost:3000/` and you'll see the `Search` Client Component successfully rendered within the `Home` Server Component and default layout! - - - -Now let's capture what the user types into the search input as state, so that we can later use it to dynamically search our Xata database. Editing `app/search.tsx`, use the `useState()` React hook to connect the value of the search input to a piece of state called something like `searchTerm`: - -```tsx -// app/search.tsx - -'use client'; - -import { useState } from 'react'; - -export default function Search() { - const [searchTerm, setSearchTerm] = useState(''); - - return ( -
- setSearchTerm(e.target.value)} /> -
- ); -} -``` - -Now that we've got a functioning search input connect to our app state, we need to actually send the search terms to Xata to find relevant games! - -However, we can't directly call Xata from the client side, because that would potentially expose the secret API key Xata uses to access our database (which we set up during `xata auth login` and added to our app's `.env` file during `xata init`). So we'll need to set up an API route on our server to fetch the data from Xata, and then fetch from that API endpoint from our client component. - -### Create an API route to return search results - -The new `app/` directory will eventually support API routes, but as of the time of writing it doesn't yet do so. In the meantime, we can [still use](https://beta.nextjs.org/docs/upgrade-guide#migrating-from-pages-to-app) the old `pages/api/` path structure to create API routes that will seamlessly mesh with the routes in our `app/` directory. This is showcased by `create-next-app` with the automatically generated `pages/api/hello.ts`, which demonstrates how to handle an API request and return a response with JSON data: - -```tsx -// pages/api/hello.ts (generated by create-next-app) - -// Next.js API route support: https://nextjs.org/docs/api-routes/introduction -import type { NextApiRequest, NextApiResponse } from 'next'; - -type Data = { - name: string; -}; - -export default function handler(req: NextApiRequest, res: NextApiResponse) { - res.status(200).json({ name: 'John Doe' }); -} -``` - -Let's repurpose this file for the `/api/search` endpoint we need. Rename the file to `pages/api/search.ts` and navigate to `localhost:3000/api/search` to see it returning the dummy "John Doe" data. - -Now, we can finally use the `xata.search.all(...)` code snippet we copied from the Xata dashboard earlier to retrieve data based on a given search term, boosting by the game's rating. - -We can use the Next [request helper](https://nextjs.org/docs/api-routes/request-helpers) `req.query` to capture the query parameters of the request, so that we can pass the search term to our endpoint via a `term` parameter like so: `/api/search?term=mario` - -Each object in the `records` array returned by `xata.search.all()` has the shape `{ table: "games", record: { id, name, console, ... }}`, but since we don't need the `table` name, extract just the `record` of each object by mapping over the array, and send the resulting array as JSON data in the response. - -Your finished `search.ts` file should look something like this: - -```tsx -// pages/api/search.ts -import type { NextApiRequest, NextApiResponse } from 'next'; -import { getXataClient } from '../../util/xata'; - -// Instantiate the Xata client -const xata = getXataClient(); - -// Make the function async to await the search data -export default async function handler(req: NextApiRequest, res: NextApiResponse) { - // Parse the 'term' query parameter from the request - const term = req.query.term as string; - - // Paste the code snippet copied from the Xata search dashboard - // passing in the search term from the request - const { records } = await xata.search.all(term, { - tables: [ - { - table: 'games', // Search the 'games' data - // Prioritize games with a higher 'totalRating' - boosters: [{ numericBooster: { column: 'totalRating', factor: 2 } }] - } - ], - fuzziness: 0, - prefix: 'phrase' - }); - - // Extract the `record` property from the { table, record } objects returned by the search - res.status(200).json(records.map((r) => r.record)); -} -``` - -Navigate to your new & improved search endpoint to try out different search terms and verify it's working! For example `http://localhost:3000/api/search?term=batman`. - -Now all that remains is to connect our client side code to the new search API route, so that our users can search games from the UI. - -### Fetch data from the client - -The React and Next teams are working on some big changes to our current patterns for [fetching data](https://beta.nextjs.org/docs/data-fetching/fundamentals) on the server & client. At the time of writing, these new patterns are quite experimental and not _quite_ at the stage where developers can fully take advantage of them. - -As we saw earlier in our `games/[gameID]/page.tsx` component, we can use `await` to fetch data in React Server Components, but it isn't supported in Client Components. Instead, React's new `use()` hook is intended to bring `await`-like functionality to Client Components, so in the future we should be able to asynchronously fetch data with `use()` in a Client Component as described in the [Next docs](https://beta.nextjs.org/docs/data-fetching/fetching#example-fetch-and-use-in-client-components) and [React's promises RFC](https://github.com/acdlite/rfcs/blob/first-class-promises/text/0000-first-class-support-for-promises.md#example-use-in-client-components-and-hooks). - -However, at the time of writing, the caching functionality that's a necessary complement to `use()` has [not been fully implemented](https://github.com/vercel/next.js/issues/42180#issuecomment-1296302329) in React/Next yet, and the intended `use()` pattern seems to create an infinite loop of rerenders and network requests in our Client Component. - -So while we eagerly await (ha!) full use (ha!!) of `use()`, we still need a way to fetch search results from our API endpoint and use it to render a basic list of games (each linking to its corresponding `/games/[gameID]` page). For now, we can do this by fetching data within a `useEffect()` and capturing the results with `useState()`, like so: - -```tsx -// app/search.tsx - -'use client'; - -import { useEffect, useState } from 'react'; -import Link from 'next/link'; - -// fetch() is not yet supported inside Client Components -// so we wrap it in a utility function -async function getData(term: string) { - const res = await fetch(`/api/search?term=${term}`); - return res.json(); -} - -export default function Search() { - const [searchTerm, setSearchTerm] = useState(''); // Track the search term - const [games, setGames] = useState([]); // Track the search results - - useEffect(() => { - if (searchTerm) { - // Update the games array once data has loaded - getData(searchTerm).then((results) => setGames(results)); - } else { - // Reset games if the search term has been cleared - setGames([]); - } - }, [searchTerm]); - - return ( - <> -
- setSearchTerm(e.target.value)} /> -
- {games.map(({ id, name }) => { - // Render a basic link to the info page for each game - return ( -
- -

{name}

- -
- ); - })} - - ); -} -``` - -Now our search app, while basic, is complete! Reload the home page at `localhost:3000/` and enjoy searching through thousands of retro games. - - - -## Recap & next steps - -In the process of building our app to search retro games data, we've covered a lot of ground! We learned how to: - -- Import CSV data into Xata -- Set up the Xata SDK in a Next project -- Use the new `app/` directory in Next.js 13 -- Build React Server and Client components in `app/` -- Perform full-text search with boosting via the Xata SDK -- (Almost) use the new `use()` hook to fetch data from Client Components - -There is still a lot more Xata functionality we haven't had time to explore. Check out [search-retro-games.vercel.app](https://search-retro-games.vercel.app) for an enhanced version of this [project](https://github.com/vakila/search-retro-games) featuring: - -- [Filtering](https://xata.io/docs/sdk/filtering) search results by console -- [Aggregating](https://xata.io/docs/sdk/aggregate) to count the total number of games -- Debounced search input to avoid over-fetching -- More details in search results & game pages - -Now let's take a break from all this coding, and go play some retro games! - -_Thanks very much to [Xata](https://xata.io) for sponsoring this work and to retro gaming queen [Sara Vieira](https://github.com/saravieira) for making it possible!_ diff --git a/securely-querying-your-database.mdx b/securely-querying-your-database.mdx deleted file mode 100644 index d3fe3212..00000000 --- a/securely-querying-your-database.mdx +++ /dev/null @@ -1,101 +0,0 @@ ---- -title: 'Securely querying your database with Xata' -description: 'Connect to a Xata database and fetch data from client-side apps without exposing security keys.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/securely-querying-your-database-on-xata.png - alt: Xata -author: Atila Fassina -date: 10-24-2022 -tags: ['security'] -published: true -slug: securely-querying-your-database-on-xata ---- - -With a database access key being sent out from the browser, your database is readable and writable by anyone. They may read the key and therefore use the key to create malicious connections to your database. These actions (attacks) are called **exploits**, and the data retrieved from such exploits are called **leaks**. Yes, like the famous ā€œdata leaksā€ we hear about in the news. Whether your leaked app makes to the news or not will depend on how big and influential your user-base is, but the problem is still the same. Someone with the key can even delete all of your data if they so choose. - -When building an app, a common pattern is to build an API (Application Program Interface) that will work as a bridge between your database and the client-side app. Said API will expose a certain number of endpoints depending on how it is designed. For example: - - - -This architecture is not only common, it's a requirement. It happens because a database requires **access keys**. As anything valuable, user data must be stored behind locked doors. Wherever the query originates from, must add these **access keys** (often referred to as **access tokens** as well) to the request. This is why any communication with a database must happen in a **private connection**. - -A browser is a **public** interface. Whatever data and code an app has in the browser is **exposed the public**. Now connect the dots: if everything on the browser is available to the public, and a query (request) goes from the browser straight to the database, what happens to the secure key it must carry to actually retrieve the data? You guessed it: the key is readable, as said in security contexts, it is **exposed**. - -## Querying Xata - -To make things more ergonomic, Xata has a [Software Development Kit (SDK)](https://xata.io/docs/sdk/typescript/overview) we call [client-ts](https://github.com/xataio/client-ts). Under the hood, there are many best practices we enforce. It also generates TypeScript types for your data request and responses based on your schema - that is the Object Relational Mapping (ORM) side of our SDK. - -But arguably, the most important aspect of our SDK is that it will securely query Xata using your environment variables (`process.env`) the `XATA_API_KEY`, which is the key required to query your database. There multiple different ways to expose this environment variable to your running app, for example: - -- when calling the script: - -```bash -XATA_API_KEY=my_api_key node ./script/query.js -``` - -- more commonly, via the `.env` configuration file: - -```bash -XATA_API_KEY=my_api_key -``` - -When deploying your app, you must find out how to define Environment Variables to the deployment script. Each PaaS (Platform as a Service) has a different way of doing that. Have a look for examples at [CircleCI](https://circleci.com/docs/env-vars/), [Cloudflare Workers](https://developers.cloudflare.com/workers/platform/environment-variables/), [Netlify](https://docs.netlify.com/configure-builds/environment-variables/), or [Vercel](https://vercel.com/docs/concepts/projects/environment-variables). - -The `process.env` namespace is not available in browser environments so if using the SDK, you will experience _friction_ when attempting to query your Xata database from the browser. **This is intentional and by design.** Essentially, `process.env.XATA_API_KEY` will be `undefined`, and the SDK will not be able to fire your request (thus preventing you from leaking access to your database), and you will see a runtime error like: - -```bash -error Error: Option apiKey is required - at XataClient.parseOptions_fn (index.mjs?4597:3090:1) - at new _a (index.mjs?4597:3053:1) - at XataClient._createSuperInternal (_create_super.mjs?05d7:12:1) - at new XataClient (xata.codegen.ts?b4b0:32:24) - at getXataClient (xata.codegen.ts?b4b0:42:14) - at eval (stepone.js?34b1:101:47) -``` - -If on a Node.js environment, this means the SDK cannot read your API key. You can solve this by using one of the methods mentioned above to expose your access key to your runtime. If using the `.env` approach, you may need a package like [dotenv](https://www.npmjs.com/package/dotenv) to read those values. - -If on a browser environment, this error \***\*\*\*\*\***\*\*\***\*\*\*\*\***\*\*\***\*\*\*\*\***\*\*\***\*\*\*\*\***should not be ā€œfixedā€\***\*\*\*\*\***\*\*\***\*\*\*\*\***\*\*\***\*\*\*\*\***\*\*\***\*\*\*\*\***. As explained in the first section of this article: exposing your Xata API keys to a client-side runtime like a browser is a huge security issue. For more information, read our documentation about [Securely Talking to Xata](https://xata.io/docs/getting-started/installation#working-with-client-sdks). - -## Creating apps and connecting to Xata - -The path of least resistance to create an app and connect it to Xata is to use a full-stack framework such as [Next.js](https://nextjs.org/docs), [Remix](https://remix.run/docs/en/v1), [Nuxt](https://nuxtjs.org), [Gatsby](https://www.gatsbyjs.com/docs/), and others. Such apps offer a way of creating **serverless functions** within them. In Next.js for example, any files in `./pages/api` contain these functions. But what are they? In a nutshell serverless functions are pieces of code that will run only on the server, allowing developers to query their database there and thus remaining close to the app logic and more importantly, **_out of the unsafe browser environment._** - -When creating a pure client-side app ([Astro](http://astro.build), [Eleventy](https://www.11ty.dev/), [Qwik](https://qwik.builder.io/), [React](https://beta.reactjs.org/), [Solid](https://www.solidjs.com/), [Svelte](https://svelte.dev/), [Vue](https://vuejs.org/), etc.), the database request will not be able to originate from the app. Instead, refer to the initial architecture we outlined in this post: first hit an API, then the API connects to the Database. This keeps your API keys secret. - -### What are serverless functions? - -Different than a long-standing server that will continue to run even when there are no requests coming in or out, a serverless function will cease activity and go into ā€œsleepā€ whenever its execution is finished. This makes serverless functions a perfect use-case to bridge connections between your app and your database. - -Additionally, serverless providers like [AWS Amplify](https://aws.amazon.com/amplify/), [Cloudflare Workers](http://workers.cloudflare.com), [Netlify](http://netlify.com), and [Vercel](http://vercel.com) will provide developers with a no-obstacle deployment. You define your function handler (most receive 2 parameters: `request` and `response`). - -## Connecting a Serverless Function to Xata - -Creating a serverless function is basically as straightforward as creating a JavaScript file. - -```jsx -// In Next.js, this would be ./pages/api/get-data.ts - -export default function handler (request, response) { - // fetch stuff from Xata - const data = await xata.db.myDb.getMany(); - response.end(data.records); -} -``` - -To understand how to create one, and how to design your API, we have a [Vercel Serverless Function Sample](https://github.com/xataio/examples/tree/main/apps/starter-vercel-serverless-functions) in our example repository. Do not forget to create and add your API Keys to the deployment routines within your dashboard, otherwise the Xata Client will not have a key to fire requests with. - -For a more involved example, refer to the [working with client SDKs](https://xata.io/docs/getting-started/installation#working-with-client-sdks). - -## Conclusion - -We hope this has made it clearer how to interact with your Xata database from within and outside your apps. Please feel free to create an issue in our [examples repository](https://github.com/xataio/examples) if there is a specific use-case that you would like us to prioritize in showing how to address. - -Additionally, we cannot stress it enough that **we love open-source** and there is some swag waiting for everyone who ships a community sample app, or creates a PR to contribute to any of our projects (examples included!). diff --git a/semantic-search-openai-typescript-deno.mdx b/semantic-search-openai-typescript-deno.mdx deleted file mode 100644 index f296d9e2..00000000 --- a/semantic-search-openai-typescript-deno.mdx +++ /dev/null @@ -1,323 +0,0 @@ ---- -title: 'Semantic search with Xata, OpenAI, TypeScript, and Deno' -description: 'Reduce complex data and perform similarity search with Xata.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/semantic-search-openai-typescript-deno.png - alt: Xata and ChatGPT logos -author: Tudor Golubenco -date: 03-02-2023 -tags: ['search', 'ai'] -published: true -slug: semantic-search-openai-typescript-deno ---- - -At the same time we launched our [ChatGPT integration](https://xata.io/chatgpt), we also added a new vector type to Xata, making it easy to store embeddings. Additionally, we have added a new `vectorSearch` endpoint, which performs similarity search on embeddings. - -Letā€™s take a quick tour to see how you can use these new capabilities to implement semantic search. Weā€™re going to use the OpenAI embeddings API, TypeScript and Deno. This tutorial assumes prior knowledge of TypeScript, but no prior knowledge of Xata, Deno, or OpenAI. - -## What is semantic search? - -Instead of just matching keywords, as traditional search engines do, semantic search attempts to understand the context and intent of the query and the relationships between the words used in it. - -For example, letā€™s say you have the following sample sentences: - -- "The quick brown fox jumps over the lazy dog story" -- ā€œThe vehicle collided with the treeā€ -- ā€œThe cat growled towards the dogā€ -- "Lorem ipsum dolor sit amet, consectetur adipiscing elitā€ -- ā€œThe sunset painted the sky with vibrant colorsā€ - -If you search for ā€œsample text in latinā€, traditional keyword search wonā€™t match the ā€œlorem ipsumā€ text, but semantic search will (and weā€™re going to demo it in this article). - -Similarly, if you search for ā€œthe kitty hissed at the puppyā€, semantic search will see that the phrase has the same meaning as ā€œthe cat growled towards the dogā€, even if they use none of the same words. Or, for another example, ā€œvanilla skyā€ should bring up the ā€œThe sunset painted the sky with vibrant colorsā€ sentence. Pretty cool, right? This is now quite possible thanks large-language models and vector search. - -## A quick intro to embeddings - -From a data point of view, embeddings are simply arrays of floats. They are the output of ML models, where the input can be a word, a sentence, a paragraph of text, an image, an audio file, a video file, and so on. - - - -Each number in the array of floats represents the input text on a particular dimension, which depends on the model. This means that the more similar the embeddings are, the more ā€œsimilarā€ the input objects are. Iā€™ve put ā€œsimilarā€ in quotes because it depends on the model what kind of similar it means. When it comes to text, itā€™s usually about ā€œsimilar in meaningā€, even if different words, expressions, or languages are used. - -Reducing complex data to an array of numbers representing its qualities turns out to be very useful for a number of use cases. Think of reverse image search, recommendations algorithms for video and music, product recommendations, categorizing products, and so on. - -## Vector type in Xata - -If you want to follow along, start with steps: - -- Sign up or sign into Xata [here](https://app.xata.io/signin) (the usage from this tutorial fits well within the free tier, so you donā€™t need to set up billing) -- Create a database named `vectors` -- Create a table named `Sentences` -- Add two columns: - - `sentence` of type string - - `embedding` of type vector. Use 1536 as the dimension - - - -When you are done, the schema should look like this: - - - -## Initialize the Xata project - -To get ready for running the typescript code, install the Xata CLI: - -```sh -npm install -g @xata.io/cli@latest -``` - -Run `xata auth login` to authenticate the CLI. This will open a browser window and prompt you to generate a new API key. Give it any name youā€™d like. - -```sh -xata auth login -``` - -Create a folder for the code: - -```sh -mkdir sentences -cd sentences -``` - -And run `xata init` to connect it to the Xata DB: - -```sh -xata init -``` - -The Xata CLI will ask you to select the database and then ask you how to generate the types. Select **Generate TypeScript code with Deno imports**. Use default settings for the rest of the questions. - - - -## Prepare OpenAI and Deno - -Create an [OpenAI account](https://platform.openai.com/signup) and generate a key. Note that you need to set up billing for OpenAI in order for to run these examples, but the cost will be tiny (under $1). - -Add the OpenAI key to the `.env` file which was created by the `xata init` command above. Your `.env` should look something like this: - -```bash -# API key used by the CLI and the SDK -# Make sure your framework/tooling loads this file on startup to have it available for the SDK -XATA_API_KEY=xau_ -OPENAI_API_KEY=sk- -``` - -Install the Deno CLI. See [this page](https://deno.land/manual@v1.31.3/getting_started/installation) for the various install options, on macOs with Homebrew it is: - -```sh -brew install deno -``` - -## Load data - -Itā€™s now the time to write a bit of TypeScript code. Create a `loadWithEmbeddings.ts` file top level in your project with the following contents: - -```tsx -import { Configuration, OpenAIApi } from 'npm:openai'; -import { getXataClient } from './src/xata.ts'; -import { config as dotenvConfig } from ''; - -dotenvConfig({ export: true }); - -const openAIConfig = new Configuration({ - apiKey: Deno.env.get('OPENAI_API_KEY') -}); -const openAI = new OpenAIApi(openAIConfig); -const xata = getXataClient(); - -const sentences: string[] = [ - 'The quick brown fox jumps over the lazy dog story', - 'The vehicle collided with the tree', - 'The cat growled towards the dog', - 'Lorem ipsum dolor sit amet, consectetur adipiscing elit', - 'The sunset painted the sky with vibrant colors' -]; - -for (const sentence of sentences) { - const resp = await openAI.createEmbedding({ - input: sentence, - model: 'text-embedding-ada-002' - }); - const [{ embedding }] = resp.data.data; - - xata.db.Sentences.create({ - sentence, - embedding - }); -} -``` - -Step by step, this is what the script is doing: - -- Uses `dotenv` to load the `.env` file into the current environment. This makes sure the `XATA_API_KEY` and the `OPENAI_API_KEY` are available to the rest of the script. -- Initializes the OpenAI and Xata client libraries. -- Defines the test data in the `sentences` array. -- For each `sentence` , calls the [OpenAI embeddings API](https://platform.openai.com/docs/guides/embeddings) to get the embedding for it. -- Inserts a record containing the sentence and the embedding into Xata. - -Execute the script like this: - -```sh -deno run --allow-net --allow-env --allow-read --allow-run ./loadWithEmbeddings.ts -``` - -If you visit the Xata UI now, you should see the data loaded, together with the embeddings. - - - -## Run search queries - -Letā€™s write another simple script that performs a search based on an input query. Name this script `search.ts`. - -```tsx -import { Configuration, OpenAIApi } from 'npm:openai'; -import { getXataClient } from './src/xata.ts'; -import { config as dotenvConfig } from ''; - -dotenvConfig({ export: true }); - -const openAIConfig = new Configuration({ - apiKey: Deno.env.get('OPENAI_API_KEY') -}); -const openAI = new OpenAIApi(openAIConfig); -const xata = getXataClient(); - -if (Deno.args.length !== 1) { - console.log('Please provide a search query'); - console.log("Example: deno run --allow-net --allow-env search.ts 'the quick brown fox'"); - Deno.exit(1); -} - -const query = Deno.args[0]; -const resp = await openAI.createEmbedding({ - input: query, - model: 'text-embedding-ada-002' -}); -const [{ embedding }] = resp.data.data; - -const results = await xata.db.Sentences.vectorSearch('embedding', embedding); - -for (const result of results) { - console.log(result.getMetadata().score, '|', result.sentence); -} -``` - -Here is whatā€™s going on in the script: - -- The beginning is similar as for the previous script, using `dotenv` to load the `.env` file and then initializing the client libraries for OpenAI and Xata -- Read the query as the first argument passed to the script -- Use the [OpenAI embeddings API](https://platform.openai.com/docs/guides/embeddings) to create embeddings for the query -- Run a vector search using the Xata [Vector Search API.](/docs/sdk/vector-search) This finds vectors that are similar to the provided embedding -- Print the results, together with the similarity score - -To run the script, execute it like this: - -```sh -$ deno run --allow-net --allow-env --allow-read --allow-run \\ - ./search.ts 'sample text in latin' - -1.8154079 | Lorem ipsum dolor sit amet, consectetur adipiscing elit -1.7424928 | The quick brown fox jumps over the lazy dog story -1.7360129 | The cat growled towards the dog -1.7311659 | The sunset painted the sky with vibrant colors -1.7038174 | The vehicle collided with the tree -``` - -As you can see, searching for ā€œsample text in latinā€ results in the ā€œLorem ipsumā€ text as we hoped. You can also try some variations, for example, ā€œsample sentenceā€ still brings the ā€œLorem ipsumā€ one as the top result: - -```sh -$ deno run --allow-net --allow-env --allow-read --allow-run \\ - ./search.ts 'sample sentence' - -1.805396 | Lorem ipsum dolor sit amet, consectetur adipiscing elit -1.7715557 | The quick brown fox jumps over the lazy dog story -1.7608802 | The sunset painted the sky with vibrant colors -1.7573793 | The cat growled towards the dog -1.7493906 | The vehicle collided with the tree -``` - -The scores on the left are numbers between 0 and 2 which indicate how close each sentence is to the provided query. If you run with a sentence that exits in the data, youā€™ll get a score very close to 2: - -```sh -$ deno run --allow-net --allow-env --allow-read --allow-run \\ - ./search.ts 'The quick brown fox jumps over the lazy dog story' - -1.9999993 | The quick brown fox jumps over the lazy dog story -1.8063612 | The cat growled towards the dog -1.769694 | Lorem ipsum dolor sit amet, consectetur adipiscing elit -1.7673006 | The vehicle collided with the tree -1.7605586 | The sunset painted the sky with vibrant colors - -``` - -Now letā€™s try the ā€œvanilla skyā€ query: - -```sh -$ deno run --allow-net --allow-env --allow-read --allow-run \\ - ./search.ts 'vanilla sky' - -1.8137007 | The sunset painted the sky with vibrant colors -1.7584264 | Lorem ipsum dolor sit amet, consectetur adipiscing elit -1.7579571 | The quick brown fox jumps over the lazy dog story -1.7505519 | The vehicle collided with the tree -1.738231 | The cat growled towards the dog -``` - -Bingo, top result matches what we expected. - -Another one to try: - -```sh -$ deno run --allow-net --allow-env --allow-read --allow-run \\ - ./search.ts 'The car crashed into the oak.' - -1.907266 | The vehicle collided with the tree -1.7763984 | The cat growled towards the dog -1.7755028 | The quick brown fox jumps over the lazy dog story -1.7745116 | The sunset painted the sky with vibrant colors -1.7570261 | Lorem ipsum dolor sit amet, consectetur adipiscing elit - -``` - -Again, the sentence with the same meaning shows up at the top with a score close to 2. - -## Conclusion - -The large-language-models are powerful tools that open up new use cases. Semantic search is one of these use cases, and weā€™ve seen how the Xata vector search can be used to implement it. You can also use it to build recommendation engines, or finding similar entries in a knowledge-base, or questions in a Q\&A website. - -If youā€™re running this tutorial, please join us on the Xata [Discord](https://xata.io/discord) and let us know what you are building! diff --git a/server-actions-databases-data-handling.mdx b/server-actions-databases-data-handling.mdx deleted file mode 100644 index fe7e53b6..00000000 --- a/server-actions-databases-data-handling.mdx +++ /dev/null @@ -1,180 +0,0 @@ ---- -title: 'Server Actions, databases, and the future of data handling' -description: 'Check out how Xata is improving data handling in modern apps using React advancements' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/server-actions-data-handling.png - alt: 'Server Actions, databases, and the future of data handling' -author: Atila Fassina -date: 06-21-2023 -tags: ['serverless', 'server'] -published: true -slug: server-actions-data-handling ---- - -At Xata we're focused on improving how modern applications handle data. So naturally, we're excited about the future of React. As a result of Concurrency in React gaining momentum and React Server Components becoming more widely understood, the data fetching story has made significant strides in terms of improvement. - -As developers, we're experiencing an exciting shift in how data integrates into our applications. Using the "Fetching then Rendering" pattern in our components provides a more enjoyable experience for developers and makes it easier to bring data into an application, resulting in a smoother overall experience. - -Notice we're querying the database straight from the component rendering function. That's amazing! On top of that, we can even wrap our components into a [suspense boundary](https://react.dev/reference/react/Suspense#usage) and let data stream as necessary. - -```tsx -export async function Movie({ slug }) { - const movie = await xata.db.movies.filter({ id: slug }).getFirst(); - - return ( - <> -
- {movie.title} -

{movie.title}

-
-
-

{movie.overview}

-
- - ); -} -``` - -Check out the above code snippet in our [XMDB template](https://xata.io/docs/examples/xmdb) - -```tsx -// app/page.tsx -async function SERP() { - searchParams -}) => { - return ( -
- }> - - -
- ) -} -``` - -While inside `SearchResult`, we can fetch, await, and render. Additionally, we can even pass the pending promise as a `prop`. The [data-fetching story](https://nextjs.org/docs/app/building-your-application/data-fetching/fetching#static-data-fetching) is extremely versatile and can adapt to business logic instead of writing logic to contort framework specific rules, abstractions, and conventions. We write our logic and we can make it perform in the best way possible without much of a workaround. - -## Acting on server - -One could argue that apart from the granularity of performing data-fetching on a per-component basis, the story was well defined before. We had similar capabilities previously (albeit with the integration of additional libraries or tools), but we could achieve the same results. The missing piece, however, was having this functionality within the framework, which unlocks the mutation story. This aspect has varied greatly between different tech stacks and poses an important decision that teams must make, with plenty of room for mistakes. - -Remix tackles the mutation story on a per-view basis, using action functions that allow for the co-location of fetching and posting data with the render logic like never seen before. The framework astutely determines user intentions based on the HTTP method (usually `GET` or `POST`, but it also fully supports others). All we need to do is export the appropriate method in our page and set the correct method within our `
` element. - -```tsx -export const action = async ({ request }: ActionArgs) => { - const xata = getXataClient(); - const { intent, id } = Object.fromEntries(await request.formData()); - - if (intent === 'delete' && typeof id === 'string') { - await xata.db.remix_with_xata_example.delete(id); - - return json({ - message: 'delete: success', - data: null - }); - } - - if (intent === 'create') { - const newItem = await xata.db.remix_with_xata_example.create(LINKS); - - return json({ - message: 'create: success', - data: newItem - }); - } - - return json({ - message: 'no action performed', - data: null - }); -}; - -const Task: TaskComponent = ({ id, title, url, description }) => { - const fetcher = useFetcher(); - - return fetcher.submission ? null : ( -
  • - - {title} - -

    {description}

    - - - - -
    • - ); -}; -``` - -Check out the official Xata template in [this Remix example](https://github.com/remix-run/examples/tree/main/xata). - -React Server Components take it a step further by introducing a higher level of granularity. Next.js leverages the `use server` directive to define Server Action methods within a component's body. This enables the definition of almost any server-exclusive logic. - -So now, the search component, that has interactivity like a regular Client Component can import a database query and fire from within its body. - -```tsx -'use client'; - -import { useTransition } from 'react'; -import { addUser } from '~/db/actions'; - -function SaveSomeData() { - let [isPending, startTransition] = useTransition(); - - return ( - <> - ... - - - ); -} -``` - -Our action will receive that data straight from the event callback and push it to our server, just like that: - -```tsx -'use server'; - -export async function addUser(data) { - const user = await xata.db.users.create(data); - - return Boolean(user) ? 'great success' : 'oops'; -} -``` - -There's plenty to go around and a lot is still under experimental flags, but the future looks bright for our data stories. It's even possible to implement optimistic transitions within your component with a `useOptimistic` hook. - -## Querying inside components - -Of course, this is not a silver bullet, and while we are celebrating with **we can**, we still need to be aware and conscious about whether (and when) we should or should not use this. As Sebastian MarkbĆ„ge himself said: - -["DB queries in your components/actions isn't really meant to be how the whole industry does everything, but I think we forgot how much it helps for prototyping"](https://twitter.com/sebmarkbage/status/1655291879682867202?s=20) - -We have another tool at our disposal and this is going to free up a lot more mental real estate to think about user-centric experiences. Querying the database from within your component's logic won't magically make your app better or faster by itself, but it will allow you to be more efficient, productive, and intentional about the time you spend building features and value to your users. And, isn't that what the Developer Experience is all about? Empowering developers to build better features by facilitating solutions! - -## Full-stack solutions and then some - -So now that we can jump in and out of our servers from within our components, and we can move the network boundary (the line between server and client) as it suits us - it's time to start leveraging those solution into unblocking more complex use-cases. - -As the needle moves away from unblocking user queries, at Xata we immediately started thinking about making developers more productive, now. That's why our latest launch has been around the **Development Workflow**. - -If a project is connected to a database, once the schema is branched out, it will match the Xata Branch to the GitHub Branch, if they have the same name, and thus creating a Preview Deployment that connects directly to that migration. Once your Pull Request is merged, Xata will automatically trigger a schema migration (with zero downtime!) and make sure your schema is consistent with the code. - -## An exciting future - -As our data story sits front and center within our favorite framework's development, we can look ahead and brainstorm new ideas and features. Branch out our data, create quick proof-of-concepts, test, iterate, and merge everything in levels of efficiency that weren't possible before. - -The future is exciting for web development and serverless, and we are absolutely here for it! - -How about you? What makes you the most excited about all these upcoming changes to our stack landscape? Let us know on [Twitter](http://twitter.com/xata), or join us on [Discord](xata.io/discord)! diff --git a/travel-application-development-guide.mdx b/travel-application-development-guide.mdx deleted file mode 100644 index 8808a1a1..00000000 --- a/travel-application-development-guide.mdx +++ /dev/null @@ -1,1592 +0,0 @@ ---- -title: 'Guide to developing a travel application: Xata, Next.js, Tailwind CSS, & Vercel' -description: 'Learn how to create a travel dashboard application using Xata, Next.js, Tailwind CSS, and Vercel to help you track your adventures and organize your travel experiences.' -image: - src: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/travel-app-guide/travel-app.png - alt: 'Guide to developing a travel application: Xata, Next.js, Tailwind CSS, & Vercel' -author: Teri Eyenike -date: 06-06-2024 -tags: ['tutorial'] -published: true -slug: travel-application-development-guide -ogImage: https://raw.githubusercontent.com/xataio/mdx-blog/main/images/travel-app-guide/travel-app.png ---- - -In this step-by-step guide, youā€™ll create a full-stack application using the CRUD technique and a Xata database to display a landing page detailing travel data, including destination details and trip notes. Youā€™ll learn how to: - -- Set up a Xata database -- Create a schema with different column types -- Handle form submissions with Next.js -- Handle client-side image uploads -- Create dynamic routes with Next.js - -![Travel application landing page](/images/travel-app-guide/landing-page.png) - -## Before you begin - -### Prerequisites - -Before you begin, youā€™ll need to do the following: - -- Have an Xata account [Sign up for a free](https://app.xata.io/signin?mode=signup) -- Install [Node.js](https://nodejs.org/en/download) 18 or later -- Create a [Vercel](https://vercel.com/) - -### Tech Stack - -The following technologies are used in this guide: - -| Technology | Description | -| ---------------------------------------- | --------------------------------------------------------------------------- | -| [Xata](https://xata.io/) | Serverless Postgres database platform for scalable, real-time applications. | -| [Next.js](https://nextjs.org/) | Framework for creating high-quality full-stack web applications. | -| [Vercel](https://vercel.com/) | A cloud-based frontend hosting and deployment platform. | -| [Tailwind CSS](https://tailwindcss.com/) | A utility first CSS framework for styling beautiful designs. | - -## Setting up a Xata database - -For setup and configuration, create a new database by clicking on **+ Add database**. - -![Adding a database](/images/travel-app-guide/adding-db.jpeg) - -After creating the database, a **table** is designed to store the content of the trips and destinations. - -![Creating a table](/images/travel-app-guide/creating-table.jpeg) - -On the left sidebar, click **Schema** to add another table called notes that links as many notes (one-to-many relationship) as possible to a trip. This table will allow users to upload photos of their trips and add text. The schema will look like this: - -![Schema view for adding additional tables](/images/travel-app-guide/schema-view-tables.jpeg) - -## Creating the schema - -Let's begin with the trips table to add columns to the table. - -Below the schema of each table, click on **+ Add a column** that opens a dialog box for the option field type. Select the appropriate field type and set city, country, start, and end columns to the trips table with the String type. - -![Adding column types](/images/travel-app-guide/adding-column-types.jpeg) - -Then, head over to the second table notes and do something similar. Create and set the columns name, and type to the String type while the description stores long-form text and characters as Text type. - -For the rating column name, set it to the Integer type. - -![Adding a rating column](/images/travel-app-guide/adding-rating-column.jpeg) - -Proceed to create a column img, which accepts the File type and allow users see the displayed image when the Make files public by default option is checked. - -![Adding an img column](/images/travel-app-guide/adding-img-column.jpeg) - -The last step is linking to the trips table, forming the relationship between the tables. Create a column as trip, and select the Link to table option type. Also, check the Unique attribute to prevent duplicate entries in the table. - -The final schema view should look like this below: - -![Final schema view](/images/travel-app-guide/final-schema-view.jpeg) - -## Setting up your project - -Run the CLI command below to create the Next.js boilerplate for the project. - -```bash -npx create-next-app@latest -``` - -On completion, navigate into its directory and run the development server to start the application on `http://localhost:3000`. - -```bash -cd track-trips -npm run dev -``` - -## Configure Xata with Next.js - -To configure the newly created app with Xata, we need to install the Xata CLI globally with the command: - -```bash -npm install @xata.io/cli -g -``` - -If this is your first time integrating or using Xata, you can authorize or authenticate the Xata CLI with your associated logged-in account: - -```bash -xata auth login -``` - -![Authorizing the Xata CLI](/images/travel-app-guide/authorizing-cli.jpeg) - -Now, initialize and connect your database locally with the Xata CLI command: - -```bash -xata init -``` - -This command configures a new project, creating a new database. - -A prompt shows some CLI questions about integrating the instance with Next.js, which creates extra files within the project folder. - -![Xata CLI integration with Next.js](/images/travel-app-guide/xata-next-integration.jpeg) - -## Creating the landing page - -Next.js is an excellent tool for creating UIs, and this applicationā€™s home page will provide information to guide users so itā€™s a key part of visualizing this information. - -In the app directory, create a route group in parenthesis (main) for personalized layouts. Within the folder, add the layout.js and page.js files. - -```js -// File: app/(main)/layout.js -import Navbar from '@/components/Navbar'; -import Footer from '@/components/Footer'; -import { Inter } from 'next/font/google'; -import '../globals.css'; - -const inter = Inter({ subsets: ['latin'] }); - -export default function RootLayout({ children }) { - return ( - - -
    - - {children} -
    -
    - - - ); -} -``` - -Ensure the `{children}` are rendered between the two components, `` and `