+ Copyright notice:
+ All articles on this website, unless otherwise stated, adopt CC BY-NC-ND 4.0
+ reprint policy. If reproduced, please indicate source!
+
Amazon Web Services (AWS) offers a wide range of storage solutions to cater to the diverse needs of businesses and developers. Two popular options are Amazon Elastic Block Store (EBS) and Amazon Elastic File System (EFS). While both provide storage capabilities within the AWS ecosystem, they serve different purposes and have distinct characteristics. In this blog post, we will explore the key differences between AWS EBS and AWS EFS to help you make an informed choice for your specific use case.
-
Amazon Elastic Block Store (EBS)
+
Amazon Elastic Block Store (EBS)
AWS EBS
Amazon Elastic Block Store (EBS) is a block-level storage service that provides durable and high-performance storage volumes that can be attached to Amazon Elastic Compute Cloud (EC2) instances..
Here are some of the main characteristics and use cases of AWS EBS.
-
Block Storage
+
Block Storage
EBS provides block storage, which means it is best suited for scenarios where you need to store data at the block level, such as databases and applications that require direct access to disk devices.
-
Low-Latency Performance
+
Low-Latency Performance
EBS volumes offer low-latency, high-throughput performance, making them ideal for I/O-intensive workloads where rapid data access is crucial.
-
Data Persistence
+
Data Persistence
EBS volumes are persistent, meaning the data stored on them remains intact even when the associated EC2 instance is stopped or terminated. This is useful for maintaining critical data.
-
Availability and Redundancy
+
Availability and Redundancy
EBS volumes can be replicated within a specific Availability Zone (AZ) for redundancy, but they are not natively designed for cross-AZ or cross-region redundancy. For cross-AZ redundancy, you need to set up additional configurations.
-
Cost
+
Cost
You pay for the provisioned capacity of the EBS volume, regardless of whether it is in use or not. This can make it cost-effective for certain use cases but might require careful capacity planning.
-
Amazon Elastic File System (EFS)
+
Amazon Elastic File System (EFS)
AWS EFS
Amazon Elastic File System (EFS) is a fully managed
Here are the key attributes and use cases of AWS EFS:
-
File Storage
+
File Storage
EFS offers file-level storage, making it suitable for scenarios where multiple instances need shared access to the same data, such as web applications, content management systems, and shared repositories.
-
Scalability
+
Scalability
EFS is designed to scale automatically as your storage needs grow. It can handle a dynamic number of EC2 instances concurrently, making it a great choice for applications with varying workloads.
-
Cross-AZ and Cross-Region
+
Cross-AZ and Cross-Region
EFS provides built-in redundancy and can be accessed across multiple Availability Zones and even across regions. This ensures high availability and disaster recovery capabilities.
-
Pay-as-You-Go
+
Pay-as-You-Go
With EFS, you pay for the storage capacity you use, making it a cost-effective option for applications with fluctuating storage requirements.
-
Simplified Management
+
Simplified Management
EFS takes care of the underlying infrastructure and scaling, allowing you to focus on your applications without worrying about managing storage hardware.
-
Choosing Between EBS and EFS
+
Choosing Between EBS and EFS
To decide between AWS EBS and AWS EFS, consider the following factors:
-
Use Case
+
Use Case
Determine whether your application requires block-level storage (EBS) or file-level storage (EFS).
-
Performance Requirements
+
Performance Requirements
Assess your performance needs. EBS is often preferred for high-performance workloads, while EFS offers good performance for a wide range of applications.
-
Scalability
+
Scalability
Consider whether your storage needs are likely to scale over time. EFS is designed for easy scalability.
-
Availability and Redundancy
+
Availability and Redundancy
If high availability and redundancy are crucial, EFS may be a better choice due to its built-in cross-AZ and cross-region capabilities.
-
Cost
+
Cost
Evaluate your budget and cost considerations. EBS charges are based on provisioned capacity, while EFS charges are based on actual usage.
-
Conclusion
+
Conclusion
In summary, AWS EBS and AWS EFS are both valuable storage services within the AWS ecosystem, but they serve different purposes and have distinct characteristics. EBS is ideal for block-level storage with high-performance requirements and provides data persistence within a single Availability Zone. On the other hand, EFS is designed for scalable file-level storage. Your choice between the two should be based on your specific use case and requirements. Understanding these differences is crucial for optimizing your AWS storage strategy and ensuring the best performance and cost-efficiency for your applications.
]]>
-
-
-
-
- <p>在数据分析中，<code>方差（Variance）</code>、<code>标准差（Standard Deviation）</code>和<code>异常值（Outliers）</code>是分析数据分布和变异性的重要统计工具。理解这些概念，并能够有效地应用它们，对于<c
-
-
-
-
-
-
-
-
-
-
-
-
-
Introduction to LangChain: Make AI Smarter and Easier to usehttps://stonefishy.github.io/2024/11/12/introduction-to-langchain-make-ai-smarter-and-easy-to-use/2024-11-12T13:49:12.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.361Z
Have you ever wondered how some apps and websites can have conversations with you, answer your questions? Many of these apps use artificial intelligence (AI), like the chatbot you might use to ask questions or get advice. But creating these smart systems can be tricky. This is where a tool called LangChain comes in to help!
LangChain is a framework that makes it easier for developers to build applications that use AI models, like chatbots or smart helpers. In this blog, we’re going to explain what LangChain is, how it works, and why it’s useful for making AI apps.
What is LangChain?
LangChain is a tool for developers that helps them build applications using large language models (LLMs)—the same kind of AI that powers chatbots, writing assistants, and more. LLMs can understand and generate text in a way that sounds like a real person. However, using these models to make powerful apps can be complicated. LangChain makes it easier by offering ready-made building blocks to connect these models to other tools, data, and even databases.
Think of LangChain like a set of Lego blocks that you can use to build cool things with AI. It saves developers time by giving them ready-made pieces to use, rather than having to create everything from scratch.
Features of LangChain
Let’s break down some of the cool features LangChain offers and how they help developers make smarter apps.
LangChain Features
Chains: Putting Multiple Steps Together Imagine you have a robot that can help with math homework. The robot might need to do multiple things to solve a problem. First, it could look up the math formula, then solve the problem, and finally explain the answer. In LangChain, these steps are called chains.
A chain is a sequence of actions where each step depends on the previous one. For example, you could create a chain where:
First, the app asks the AI to pull data from a website. Then, it uses that data to answer a question. Finally, it summarizes the answer for the user.
Prompt Management: Talking to AI the Right Way When you talk to an AI, how you ask your question or give your instruction is really important. That’s called a prompt. LangChain helps developers make the best prompts by letting them create templates. These templates let developers easily change certain parts of the prompt without having to rewrite it every time.
For example, if you wanted to ask the AI to summarize a story, you could have a prompt like this:
“Please summarize the following story: {story}”
In this template, {story} is a placeholder that can be replaced with any story you want the AI to summarize.
Agents: Letting AI Decide What to Do Next Sometimes, a smart system needs to decide what to do next based on the information it gets. For example, if you ask an AI about the weather, it might decide to pull the latest weather data from the internet. This decision-making is done by agents.
An agent is like a helper that looks at the information it gets and chooses the best action. LangChain helps developers build agents that can make these decisions automatically.
Memory: Remembering What Happened Before Have you ever talked to a chatbot and then later felt like it forgot what you said earlier? That can make a conversation feel weird. LangChain helps solve this problem by letting the AI remember what was said earlier in the conversation. This feature is called memory.
or example, if you ask a chatbot for homework help and then ask a follow-up question, LangChain can help the AI remember the first question and give a more useful answer based on that memory.
Integrations: Connecting to Other Tools and Websites Sometimes, an AI app needs to talk to other systems to get more information. LangChain makes this easy by letting developers connect their AI app to other tools This is like having a personal assistant that not only talks to you but also has access to tons of information online.
For example, an AI app could pull up the latest sports scores, or check the weather for you, using real-time data from the internet.
Retrieval-Augmented Generation (RAG): Getting Smarter Answers LangChain also lets AI search for information in real-time. This is called retrieval-augmented generation (RAG). It allows the AI to look up the latest data, like news stories or facts, and use that information to create smarter answers.
For example, if you ask about the latest trends in video games, the AI can search the web for the most up-to-date information and then explain it to you.
Why Do Developers Use LangChain?
There are several reasons why developers might want to use LangChain:
Makes It Easier to Build AI Apps Instead of starting from scratch, LangChain gives developers tools that speed up the process of creating AI apps. Developers can use LangChain’s building blocks to create powerful applications without needing to write everything by hand.
It’s Flexible LangChain can be used for a wide variety of apps. Whether you want to build a chatbot, a smart search engine, or an app that helps you study, LangChain has tools that make it easier to put everything together.
Saves Time Developers don’t have to spend a lot of time figuring out how to make an AI model work with a database or how to chain steps together. LangChain does much of the heavy lifting, so developers can focus on the fun and creative parts of building their apps.
It’s Open-Source LangChain is free for anyone to use and improve. It’s open-source, which means developers from all over the world can contribute to making it better. If you’re learning to code or want to help improve the tool, you can!
Real-World Examples of LangChain
LangChain is already being used in many cool ways. Here are a few examples:
Chatbots Developers can use LangChain to build chatbots that remember previous conversations and can talk to you like a real person. For example, you could create a chatbot to help you study for a test, and it would remember what you’ve learned so far.
Smart Assistants LangChain can help build systems that pull information from the internet and use AI to explain things in simple terms. For example, if you’re stuck on a science problem, an AI could look up the topic online and explain it to you in a way you understand.
Automated Content Creation Some apps use LangChain to automatically write articles or summaries. For example, a news website could use LangChain to summarize long articles or pull out the key points from reports, saving readers time.
Personalized Search Engines LangChain can be used to build search engines that don’t just give you a list of links but also summarize the best results for you. This could help you find the exact answer you need faster.
How to Get Started with LangChain
If you’re excited to try out LangChain, here’s how you can get started:
Install Python: LangChain works with Python, a programming language that’s great for beginners.
Install LangChain: You can install LangChain by running the command pip install langchain in Python.
Start Building: Once LangChain is installed, you can start building your own AI-powered applications! LangChain has tutorials and guides to help you learn how to use it.
LangChain is a super helpful tool for developers who want to build cool apps powered by AI. It makes it easier to connect different parts of an app, like databases or the web, with a language model that can understand and generate text. Whether it’s helping with homework, answering questions, or building a chatbot, LangChain is a great way to build smarter, more interactive applications.
]]>
@@ -71,7 +45,7 @@
https://stonefishy.github.io/2024/11/05/what-is-prompt-engineer/2024-11-05T14:08:09.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.361ZAs artificial intelligence (AI) models, especially large language models (LLMs) like OpenAI’s GPT series, have become increasingly sophisticated, a new role has emerged in the AI ecosystem: the Prompt Engineer. The term might sound technical or niche, but it’s actually pivotal to leveraging AI models effectively. Whether you’re interacting with AI in your personal or professional life, the quality of the interaction largely depends on how well the prompt is designed. This article will explore what a prompt engineer does, the best practices for writing effective prompts, and provide examples comparing outputs with and without a prompt engineer’s expertise.
What is a Prompt Engineer?
A Prompt Engineer is someone who specializes in crafting, refining, and optimizing prompts to ensure that AI models respond with the most relevant,accurate,and actionable information. The role requires a blend of creativity, technical understanding, and knowledge of the AI’s underlying model architecture.
In essence, the prompt engineer’s job is to “speak” the language of the AI model. Since AI models like GPT-3 or GPT-4 don’t “think” like humans, their responses depend heavily on how the question or task is framed. A prompt engineer ensures that the right context, constraints, and phrasing are in place to guide the model toward producing the most useful responses.
Why is Prompt Engineering Important?
While AI models are capable of generating human-like text and performing complex tasks, their outputs are highly sensitive to the structure of the prompt. The same AI model could provide vastly different answers depending on the way a question is asked. Prompt engineers understand this sensitivity and use it to maximize the effectiveness of the interaction with AI.
Here are some reasons why prompt engineering is important:
Maximizing output quality: Well-designed prompts improve the accuracy, relevance, and clarity of responses. Reducing errors: By properly framing a prompt, prompt engineers can help reduce misunderstandings or irrelevant responses. Efficiency: Instead of relying on trial and error to get useful responses, prompt engineers streamline the interaction process, saving time and resources. Contextuality: A good prompt will provide the necessary context for the model, ensuring that the response is in line with the user’s expectations.
The Path of a Prompt Engineer
The process that a prompt engineer follows to ensure optimal results involves several stages. Each stage builds upon the last, leading to an iterative cycle that refines both the prompt and the AI’s output. Here’s a breakdown of the typical path of a prompt engineer:
The Path of a Prompt Engineer
1.Task Understanding:
Before crafting any prompt, the first step is to fully understand the task at hand. This involves clarifying the user’s goal, determining the desired output format, and understanding the nuances of the request. A deep understanding of the problem ensures that the prompt engineer can craft a question or instruction that addresses all necessary aspects.
Example: If the task is to generate a poem, the prompt engineer will need to understand the tone, style, and subject matter required.
2.Crafting Prompts:
The next step is to craft the prompt. This involves framing the task clearly, with enough specificity to guide the AI toward the desired output. Crafting an effective prompt is not about asking a single question, but about providing the model with the right context, constraints, and direction.
Example: Instead of asking, “Write a poem,” a more specific prompt might be, “Write a rhyming poem about the beauty of autumn, focusing on imagery and feelings of nostalgia.”
3.Prompt Alignment:
At this stage, the prompt must be aligned with the intended outcome. This means considering the AI model’s strengths and limitations and ensuring that the prompt leads the AI to produce a response that fits the desired format, tone, and depth. The prompt should ensure the model understands the context of the task, as well as any constraints or preferences that need to be respected.
Example: For a technical article, aligning the prompt would involve ensuring the language model knows to prioritize clarity, accuracy, and technical precision.
4.Optimizing Prompt:
After alignment, the prompt may need further refinement. This step involves fine-tuning the wording, simplifying complex instructions, or narrowing down the scope to ensure that the prompt is as effective as possible. Optimization often involves making the prompt more specific and reducing ambiguity.
Example: “Write a 300-word summary of the research paper on AI ethics, emphasizing the ethical dilemmas and implications for technology companies.” This version is more optimized than a broad, vague instruction.
5.AI Model Processing:
Once the optimized prompt is provided, the AI model processes it and generates a response. This is where the model applies its underlying machine learning architecture, leveraging its training data to formulate a response.
Example: The AI will analyze the prompt, consider patterns in its training data, and produce a response based on its understanding of the language and context.
6.Generating Output:
The AI model generates the initial output based on the prompt. Depending on the AI model’s capabilities, this output may vary in length, style, accuracy, or even relevance to the task.
Example: If the task was to summarize a paper, the output might include key findings, conclusions, and references to methodology.
7.Output Refinement:
Once the output is generated, prompt engineers review and refine it. This may involve removing irrelevant information, adjusting tone, adding details, or improving clarity. In some cases, the output might need to be restructured to fit the desired format.
Example: If the AI’s response contains tangential information or lacks clarity, the prompt engineer would reword it or fine-tune the output to better align with the user’s expectations.
8.Iterative Improvement:
Finally, the process of prompt engineering is iterative. After refining the output, prompt engineers analyze the effectiveness of the response and assess how the prompt can be improved for future tasks. This leads to continuous improvement, ensuring that future prompts are even more optimized, concise, and aligned with user needs.
Example: The engineer might adjust the prompt for the next interaction to ensure more relevant details or a more focused response.
Key Skills and Tools of a Prompt Engineer
Prompt engineering requires a variety of skills:
Understanding of Language Models: A prompt engineer should have a deep understanding of how LLMs like GPT process language. Knowing their strengths and weaknesses allows for better prompt design.
Communication Skills: Effective communication is critical, as prompt engineers must be able to convey complex instructions in a way that the model can interpret clearly.
Creativity and Experimentation: Crafting effective prompts often requires trial and error, testing different phrasings and structures to see what works best.
Analytical Thinking: Understanding how different types of inputs influence the model’s outputs and iterating to improve results.
In addition to these skills, prompt engineers also use tools to test and refine their prompts. For instance, platforms like OpenAI’s Playground allow users to experiment with various prompts in real-time, while more advanced professionals might leverage APIs to automate or scale their prompt engineering work.
Best Practices for Prompt Engineering
There are several strategies that a prompt engineer can employ to get the most out of a language model. Below are some of the best practices:
Be Specific and Clear: Ambiguous prompts can confuse AI models, leading to vague or incorrect responses. Make sure the prompt is clear and as specific as possible.
Example: Instead of asking, “Tell me about AI,” a more specific prompt would be, “Can you explain the difference between supervised and unsupervised learning in AI?”
Use Context Effectively: Providing context can guide the model to better understand the desired output.
Example: Instead of saying, “Write a poem,” say, “Write a rhyming poem about the beauty of autumn with a melancholic tone.”
Limit the Scope: Sometimes, less is more. Limit the scope of the prompt to avoid overwhelming the model with too much information or too many instructions.
Example: Instead of “Write an article about the importance of artificial intelligence in modern business, covering all aspects of AI from machine learning to natural language processing,” you could say, “Write a short article explaining the importance of AI in customer service.”
Test and Iterate: A prompt engineer should test various iterations of a prompt to identify the most effective structure.
Give Examples: For tasks requiring specific output formats, include an example to guide the model.
Example: If you want a bulleted list, you could say, “List the steps in a process to build a website. For example: Step 1: Plan the layout.”
Use Temperature and Max Tokens: Some models allow you to adjust the temperature (which controls randomness) and the max tokens (which sets a character limit) to control the output. These can be adjusted to fine-tune the model’s output.
Comparing with and without Prompt Engineering
Now let’s look at some concrete examples of how a well-crafted prompt versus a poorly constructed one can affect the outcome.
Example 1: Writing a Research Summary
Without a Prompt Engineer:
Prompt: “Summarize this research paper.”
The AI may generate a generic or overly simplistic summary, without capturing the key aspects of the paper, such as methodology, results, and conclusions.
With a Prompt Engineer:
Prompt: “Summarize the research paper titled ‘Exploring AI Ethics in Autonomous Vehicles.’ Focus on the methodology, key findings, and implications for policy. Keep the summary under 200 words.”
The AI’s response will be more targeted, concise, and aligned with the user’s expectations, providing a detailed summary that addresses the core aspects of the paper.
Example 2: Writing a Creative Story
Without a Prompt Engineer:
Prompt: “Write a story.”
The story might lack direction, coherence, or creativity, leading to a generic or even nonsensical narrative.
With a Prompt Engineer:
Prompt: “Write a short story set in a post-apocalyptic world where humans are living on Mars. The protagonist is a scientist struggling with the ethical implications of using artificial intelligence to terraform the planet. Make the tone introspective and thought-provoking.”
The story produced will be richer, more engaging, and aligned with the specific context and themes the user wanted.
Example 3: Asking for Code
Without a Prompt Engineer:
Prompt: “Write a Python function.”
The AI may generate a simple function, but it may not meet the user’s needs or lack important features such as error handling or optimization.
With a Prompt Engineer:
Prompt: “Write a Python function to validate an email address using regular expressions. The function should return True if the email is valid and False if it is invalid. It should also handle common edge cases such as missing domain names or incorrect characters.”
The AI’s response will be much more precise, including the correct implementation, error handling, and edge case considerations.
Conclusion
In the world of AI, a Prompt Engineer plays a critical role in ensuring that AI models deliver optimal results. The expertise of prompt engineers can dramatically influence the quality, relevance, and accuracy of responses from language models like GPT-4. By following best practices—such as being specific, providing context, testing different iterations, and using examples—they can significantly improve the interaction between humans and AI.
As AI continues to evolve, the role of prompt engineering will become even more important, helping users and businesses unlock the full potential of artificial intelligence. Whether it’s writing, problem-solving, or complex technical tasks, the way we interact with AI will increasingly depend on how well we craft our prompts.
]]>
@@ -103,7 +77,7 @@
https://stonefishy.github.io/2024/11/01/understanding-the-azure-openai-gpt-4-api-role/2024-11-01T10:34:01.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.361ZAzure OpenAI is a service provided by Microsoft that integrates OpenAI’s advanced language models into the Azure cloud platform. It allows developers to access and use OpenAI’s capabilities, such as natural language processing, code generation, and more, through Azure’s infrastructure.
Recently, we have deployed our first version of the OpenAI GPT-4 model into the Azure cloud platform. This model is a powerful natural language model that can generate text based on a given prompt.
What is GPT-4?
GPT-4 is a transformer-based language model that was developed by OpenAI. It is a powerful language model that can generate text based on a given prompt. It has been trained on a large dataset of text and can generate coherent and engaging text that is often considered to be the next big thing in language models.
How can I use the GPT-4 API?
To use the GPT-4 API, you need to follow these steps:
Create an Azure account.
Create a resource group.
Create a new OpenAI resource.
Generate an API key.
Use the API key to make API requests.
1. Create an Azure account
To use the GPT-4 API, you need to have an Azure account. If you don’t have one, you can create one for free by following the steps in the Azure sign-up page.
2. Create a resource group
Create a resource group to organize your Azure resources. To create a new resource group, follow these steps:
Click on the All resources option in the left-hand menu.
Search for your OpenAI resource.
Click on the resource.
Click on the Show access keys button.
Copy the Key 1 value.
5. Use the API key to make API requests
To make API requests, we need to include the API key in the request headers. Here’s an example of how to make a request to the GPT-4 API with REST styles.
1 2 3 4
curl $AZURE_OPENAI_ENDPOINT/openai/deployments/gpt-4o/chat/completions?api-version=2023-07-01-preview \ -H "Content-Type: application/json" \ -H "api-key: $AZURE_OPENAI_API_KEY" \ -d '{"messages":[{"role": "system", "content": "There are 5 classifications: Suggestion, Meanless, Compliment, Complaint, Please provide a classification for user input."},{"role": "user", "content": "Does Azure OpenAI support customer managed keys?"},{"role": "assistant", "content": "A classification word"},{"role": "user", "content": "its great and easy to use"}]}'
Place your API key and endpoint in the appropriate variables, and update which deployments model (gpt-4, gpt-4o or other models) and model version of your endpoint is using.
The Azure OpenAI also supports multiple programming languages, including Python, JavaScript, and C#. You can use the API to generate text in your preferred programming language.
GPT-4 Chat Roles
In above message parameter, you may notice thata there are three roles: system, user, and assistant. The GPT-4 API supports three chat roles. Let digger deeper into each role:
System: This role sets the context or guidelines for the conversation. It’s where you can specify instructions or constraints for how the assistant should behave throughout the interaction.
User: This role represents the input from the person interacting with the model. Any questions or prompts posed by the user fall under this role.
Assistant: This role is for the model’s responses. It contains the output generated by the assistant based on the user input and the context provided by the system.
In another word. The system role sets the context, the user role represents the input, and the assistant role contains the output.
Use system to define the conversation’s tone, behavior, or rules. Use user for all queries or statements made by the person. Use assistant for the model’s replies.
Example
Let’s say we want to classify the product feedback classification as Suggestion, Meanless, Compliment, Complaint, or Others. We can use the GPT-4 API to generate text based on the given prompt and classify the feedback.
First, we define the context or guidelines to let the assistant know what result we want to achieve. Given below content to the system role.
1
{"role": "system", "content": "There are 5 classifications: Suggestion, Meanless, Compliment, Complaint, Please provide a classification for user input."},
And, we only the OpenAI to reply me the classification word when user input is provided. So we define the assistant role as classification word.
1
{"role": "assistant", "content": "A classification word"}
Now, we can ask the user to provide the feedback and provide the user role.
1
{"role": "user", "content": "its great and easy to use"}
After the conversation, the assistant role will provide the classification word as Compliment. You will notice that there is a piece of json indicates the assistant role content value in the response. The OpenAI gpt-4o model knows “its great and easy to use” is a “Compliment” and provides the classification word as “Compliment”.
The assistant role will provide the classification word as Meanless. Because this input is not meanful for any product feedback.
Conclusion
In this article, we have explored the role of the Azure OpenAI GPT-4 API and how it can be used to generate text. We have also learned about the chat roles and how to use them to classify the product feedback.
]]>
@@ -130,7 +104,7 @@
https://stonefishy.github.io/2024/10/29/data-analysis-chart-by-generative-ai/2024-10-29T14:18:48.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.361ZIn data analysis, we often need to create charts to visualize the data by using BI tools, such as Tableau, Power BI, AWS Quicksight, or Qlik Sense. These tools allow us to create interactive and visually appealing charts, which can help us to identify patterns and trends in the data.
General Solution Architecture for Data Analysis BI Chart
The general data analysis BI chart solution architecture like below:
General Solution Architecture for Data Analysis BI Chart
.
Usually, the engieering create business chart by using BI tools after data is ETL proceseed. If the business want to see the data distribution chart, they need to ask the data engineer to create the chart. The data engineer will create the chart using BI tools and share it with the business. This may take some time and effort.
Solution Architecture for Data Analysis BI Chart by Generative AI
Think about the scenario where the business want to see the data distribution chart without the data engineer’s help. How can we create the chart without the data engineer’s help?
One way to create the data distribution chart without the data engineer’s help is to use a generative AI model. The business just need to describe what the data they want to see and want to display as which chart type. The generative AI application will create the chart for them.
The core important thing we need to let the GenAI to understand user’s natural language and generate the information which application can be use. The solution architecture like below:
AI Solution Architecture for Data Analysis BI Chart
.
The AI application will take the user’s natural language as input and generate the chart for them. The AI application will use the following steps to generate the chart:
Understand the user’s natural language and generate the chart title, chart type and chart related sql.
Connect to the database or dataset and execute the chart related sql to get the data.
Use the data to create the chart using the chart type.
The AI model could be ChatGPT, OpenAI or Claude model. The AI model will generate the chart related sql, chart type and chart title based on the user’s natural language. The AI model will use the chart type to create the chart.
For example, if the user’s natural language is “Show the distribution of the sales by product category”, the AI application will generate the chart title as “Sales Distribution by Product Category” and chart type as “Bar Chart”. The AI application will execute the following sql to get the data:
1 2 3
SELECT product_category, SUM(sales) as total_sales FROM sales_data GROUPBY product_category
Below is a demo to generate the chart for the user’s natural language base on testing sample data.
.
User input natural language:
1
"I want to see the distribution of all product categories with duplicate devices removed, and exclude the empty category, please display it in a pie chart."
.
The AI model response below base on the user’s natural language and prompts.
1 2 3 4 5
{ 'sql':"SELECT category, COUNT(DISTINCT macaddress) as device_count FROM device_demo_data WHERE category != '' GROUP BY category ORDER BY device_count DESC", 'chart': 'pie', 'title': 'Distribution of Unique Devices by Product Category' }
There is a question, how the AI model know to generate this data format for us? Actually it is because we provide the prompts to the AI model. The prompts will guide the AI model to generate the chart related sql, chart type and chart title. Below is sample prompts:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
You are a data analyst. Below is the table structure information about devices reported data. I will ask you questions, and then please generate the json data format {'sql':'','chart':'table','title':''} based on the questions I asked. Emphasize: I only need this json data format. The 'sql' value is used by AWS Athena to query and generate the chart data. The 'chart' value is the chart type, 'numeric' represents a just a number, 'table' represents a table chart, 'pie' represents a pie chart, 'bar' represents a bar chart, 'line' represents a line chart. The 'title' value is the chart title. Please remember, I only need the json string format data, don't need other sentence. CREATE EXTERNAL TABLE `device_demo_data`( `macaddress` string COMMENT '设备mac地址 / The device macaddress', `productname` string COMMENT '设备产品名称 / The device product name', `category` string COMMENT '设备产品的分类 / The device product category', `country` string COMMENT '设备所在的国家 / The country where the device is located', `region` string COMMENT '设备上传数据所在的区域 / The region where the device data is reported', `default_region` string COMMENT '设备出厂设置的默认区域 / The default region set by the device', `oneosversion` string COMMENT '设备OneOS的版本号 / The OneOS version of the device', `firmwareversion` string COMMENT '设备的固件版本号 / The firmware version of the device', `officialversion` string COMMENT '设备是否是官方的发布版本，1为官方版本， 0为非官方版本 / Whether the device is an official release, 1 for official version, 0 for non-official version', `createtime` string COMMENT '设备上报数据的日期, 数据类型是字符串，日期格式是2024-09-01，表示2024年9月1日 / The date when the device reported data, the data type is string, the date format is 2024-09-01, which means September 1, 2024' )
We tell AI model the data schema and each field means, and indicate only need the json format response with specific field. Once the AI model generate the response, we can use it to create the chart for the user’s natural language.
We can also generate the line chart base on time line.
User input natural language:
1
How many distinct devices reported every week, exclude the empty date, display the data in line graph
.
For the front-end UI to display the chart, we can use some chart library like Echarts, Hightcharts, D3.js or Chart.js. The front-end UI display the chart base on chart type and the data which queried by SQL.
All these generated charts can be added into dashboard.
.
Limitation
As you can see, the AI solution architecture is a new way to create BI chart. However, it still has some limitations. It can not generate the complex chart which like some BI tools advanced chart fucntionality. But it is still a good start to create the chart for business users.
]]>
@@ -147,10 +121,10 @@
-
-
+
+
@@ -160,7 +134,7 @@
https://stonefishy.github.io/2024/10/23/intelligent-sql-generator-base-on-spring-ai-with-aws-bedrock/2024-10-23T15:01:12.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.361ZThe generative AI is a type of artificial intelligence (AI) that can learn from data and generate new data. In this article, we will discuss how to build an intelligent SQL generator using Spring AI and AWS Bedrock. For example, the application able to provide the data sql to us after we input the natural language questions, and we can query the data by using the sql, even display the chart base on data queried.
Spring AI
The Spring AI is a project of Spring. It support for all major AI Model providers such as Anthropic, OpenAI, Microsoft, Amazon, Google, and Ollama. Model type supports such as Chart Completion, Text to Image, Text to Speech, Translation, Audio Transcription and so on. It make it easy to integrate AI models into the application.
Spring AI
AWS Bedrock
Amazon Bedrock is a fully managed service that makes FMs from leading AI startups and Amazon available via an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case. It contains Anthropic (Claude), Meta (Llama) and Stability AI models. In this blog, we will use Claude AI model of Anthropic to build our intelligent SQL generator.
AWS Bedrock
Building Intelligent SQL Generator
Assuming we’re data analyzer, we have product_sales, products and customers three tables data in mysql. And we want to query the data by input natural language instead of write specific SQL manually. We can use AI to understand user natural langauge to generate the SQL base on table schemas. Let’s get start.
Create a new Spring Boot project
Create a Spring Boot project with restful api, add the following dependencies in maven pom.xml
# config below AWS credential key, configure it in Java environments or System environments spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID} spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
Prepare prompts for AI
Since we only need AI to generate the SQL base on mysql table schema. So we need prepare prompts to AI to fully understand our requirements. Below is prompts.txt
There are 3 mysql tables schema product_sales, products, customers. CREATE TABLE product_sales ( sale_id INT AUTO_INCREMENT PRIMARY KEY, product_id INT NOT NULL, sale_date DATETIME DEFAULT CURRENT_TIMESTAMP, price DECIMAL(10, 2) NOT NULL, customer_id INT, region VARCHAR(100), FOREIGN KEY (product_id) REFERENCES products(product_id), FOREIGN KEY (customer_id) REFERENCES customers(customer_id) ); CREATE TABLE products ( product_id INT AUTO_INCREMENT PRIMARY KEY, product_name VARCHAR(100), product_category VARCHAR(100) ); CREATE TABLE customers ( customer_id INT AUTO_INCREMENT PRIMARY KEY, customer_name VARCHAR(100) ); I will ask you question, please base on table schema to generate the SQL text in single line, please note I only need full correct sql text, do not need other text or any other characters.
In above prompts, you can see it tells AI we only need SQL text base on the 3 mysql table schemas.
Core Code
Create a restful controller and pass the prompts and user input message to AI model.
In above code, we inject BedrockAnthropic3ChatModel which is Anthropic model of AWS Bedrock provided by Spring AI. We also inject PromptService which is a service to read prompts. In generate method, we read prompts from PromptService and append user input message to it. Then we call chatModel.call method to generate the SQL text.
public String getContent() { return fileContent; } }
Run the application
Run the application and test the API by sending a request with user input message. We may encounter below errors.
AWS Bedrock Model can not access
This is because the spring.ai.bedrock.anthropic3.chat.model in current Spring AI version default value is anthropic.claude-3-sonnet-20240229-v1:0. Let’s check the anthropic available Claude models in AWS Bedrock. Here we use Claude 3.5 AI model.
AWS Bedrock Anthropic Claude Model Id
Copy this model id and update the spring.ai.bedrock.anthropic3.chat.model in application.properties file. The fully updated application.properties file should be like below.
# config below AWS credential key, it also can be configure in Java environments or System environments spring.ai.bedrock.aws.access-key=${AWS_ACCESS_KEY_ID} spring.ai.bedrock.aws.secret-key=${AWS_SECRET_ACCESS_KEY}
Now, we can run the application and test the API.
Test Example 1
Input:
1
What's the total prices of product sales ?
Output:
The response will be like below.
1
SELECTSUM(price) FROM product_sales;
Postman Screenshot:
The AI model can generate the SQL text base on user input message.
Test Example 2
Input:
1
How many customers by our products? I only need unique customers.
Output:
1
SELECTCOUNT(DISTINCT customer_id) FROM product_sales
Postman Screenshot:
Test Example 3
Input:
1
I want to see the total sales prices for each product categories.
Output:
1
SELECT product_category, SUM(price) AS total_sales FROM product_sales JOIN products ON product_sales.product_id = products.product_id GROUPBY product_category;
Postman Screenshot:
Test Example 4
Input:
1
Please show me all sales data which contains price, sales date, customer name and product name and product categories
Output:
1
SELECT ps.price, ps.sale_date, c.customer_name, p.product_name, p.product_category FROM product_sales ps JOIN products p ON ps.product_id = p.product_id JOIN customers c ON ps.customer_id = c.customer_id
Postman Screenshot:
Test Example 5
Input:
1
Please show total sales prices of each product category on 2nd quarter this year
Output:
1
SELECT p.product_category, SUM(ps.price) AS total_sales FROM product_sales ps JOIN products p ON ps.product_id = p.product_id WHEREYEAR(ps.sale_date) =YEAR(CURDATE()) AND QUARTER(ps.sale_date) =2GROUPBY p.product_category
Postman Screenshot:
As you can see the AI model can generate the SQL text base on user input message. Base on this function, we can download the data from mysql or display the chart base on data queried. It’s good for business analyst to query the data by natural language.
]]>
@@ -194,7 +168,7 @@
https://stonefishy.github.io/2024/10/16/the-useeffect-of-react-runs-twice-in-development-mode/2024-10-16T16:23:44.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.361ZHave you noticed that your useEffect hook of React runs twice when the page first loads in development mode? This occurs because since React 18, it can be confusing, especially for new developers. Let’s explore why this happens and what it means.
React Hook useEffect
What is useEffect?
useEffect is a hook that allows you to perform side effects in your components. Side effects can be things like fetching data, subscribing to events, or changing the DOM. This hook takes two arguments:
A function to run your side effect.
An optional array of dependencies that tells React when to run the effect again.
In above code, the log ‘Effect has been executed’ will be printed to the console twice when the component is first render on development mode.
React useEffect runs twice in development mode
Why Does useEffect Run Twice in Development Mode?
When you run your React app in development mode, you might see that the useEffect runs twice when the component loads for the first time. This can be confusing, especially for new developers.
Reasons for the Double Execution
Strict Mode: This behavior is part of React’s Strict Mode. It purposely runs certain lifecycle methods and hooks like useEffect twice during development. This helps check if your code can handle side effects correctly.
Testing Effects: By running the effect two times, React tests if your side effects can handle being called multiple times without causing bugs. This helps catch problems early.
What Happens in Production?
The double call only happens in development mode. When you make your app for production
How to Handle the Double Execution
Here are some tips for dealing with the double execution of useEffect:
Be Careful with State Updates: If your effect updates state, make sure it’s safe to run the effect multiple times without causing issues.
Use Cleanup Functions: Always return a cleanup function from your useEffect to free up resources and avoid memory issues.
1 2 3 4 5 6 7
useEffect(() => { // Your side effect code here
return() => { // Cleanup code here }; }, [dependencies]);
Test Your Effects: Use the extra invocation to ensure that your effects work correctly.
Disable Strict Mode
It is not recommend this way, but if you want to disable Strict Mode, in React 18 you can disable Strict Mode by removing the <React.StrictMode> tag from the return statement in your root component.
In Next.js, you can disable Strict Mode by setting the following parameter in next.config.js:
1 2 3
module.exports = { reactStrictMode: false, }
Conclusion
Seeing useEffect run twice in development mode can be surprising Understanding this behavior and preparing your code for it will allow you to use React hooks effectively and build better applications.
Even though this might seem confusing at first, it’s an important part of the React development experience. Happy coding!
Recently, we’re planing to use Pulumi to manage all current existing AWS Glue Datacatalog tables which are Iceberg format. For the Iceberg tables, I have post a blog before to talk about what is iceberg and what’s feature of it. Here is post link: https://stonefishy.github.io/2020/05/23/what-is-apache-iceberg/
To manage the AWS Glue Iceberg tables with Pulumi, due to our catalog table schemas are continue changes base on requirements. We need to do some technical POC whethere the pulumi can also support to update the iceberg metadata schema as well.
Create Glue Iceberg Table
We’re using Pulumi to manage the AWS Cloud Infrastructure. Before create glue table, a glue database is indeed.
There is important thing to notice here is that we need to set open_table_format_input with iceberg_input and set metadata_operation as CREATE. This is because we want to create a new Iceberg table with new schema.
Below is glue iceberg table created screenshot. You can see the 4 fields is added in schema and table format is Apache Iceberg.
AWS Glue Table Schema Created by Pulumi
Next, let’s check the important file that is Apache Iceberg metadata file which is located in s3://xxx/pulumi_external_table_test/metadata/. Download this json file 00006-fd122b03-a7aa-42cf-8fec-001535a9fcf5.metadata.json from S3. The 4 fields are defined in metadata json file. That is good. The metadata json is created as well when creating glue table.
AWS Glue Table Iceberg metadata
Insert new data in Glue iceberg table
Let’s using AWS Athena to insert a test data in the table.
COLUMN_NOT_FOUND: Insert column name does not exist in target table: test5. If a data manifest file was generated at 's3://xxxxxx/4c346103-60d2-45ea-9813-d7060bd5efe9/Unsaved/2024/10/09/37f67a67-4604-43cb-b113-af351c363a51-manifest.csv', you may need to manually clean the data from locations specified in the manifest. Athena will not delete data in your account. This query ran against the "pulumi_database_test" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 37f67a67-4604-43cb-b113-af351c363a51
AWS Athena insert glue iceberg table failed
But when we check the Apache Iceberg metadata file again. The new field test5 is not added in the new metadata file. That’s why the insert new data with new field failed.
AWS Glue iceberg table metadata not updated
Conclusion
In Pulumi documentation. The metadata_operation of iceberg_input in open_table_format_input is only support CREATE value. It seems it only can create the iceberg metadata file when glue table created.
Pulumi API Doc
It seems this is pulumi issue. It is not updating the iceberg metadata file when the glue table schema is updated. I’ve raised a issue to pulumi, here is issue link: https://github.com/pulumi/pulumi/issues/17516. Hope this issue can be fixed soon.
The App Product User Survey Feedback Sentiment Analysis Solution is a cloud-based solution that uses AWS services to analyze user feedback and sentiment of the app product. The solution uses Amazon Comprehend to perform sentiment analysis on the feedback and Amazon S3 to store the data. The solution is designed to be scalable and cost-effective, and can be easily integrated into any app product.
Basically, our survey feedback file is Excel file that contains the user feedback of app and related application info such as OS verion and app version. The feedback text is different language from global users, so we need to translate the text into English using Amazon Translate. Besides, the feedback file is generated monthly. So, the solution will extract the feedback data from the Excel file, translate the text into English using Amazon Translate, perform sentiment analysis using Amazon Comprehend, and store the data in Amazon S3. The solution will also provide a dashboard to visualize the sentiment analysis results.
The Amazon Comprehend is a natural language processing (NLP) service that can analyze text and extract insights such as sentiment, syntax, entities, and key phrases.
Solution Architecture
Below is the high-level architecture of the solution:
AWS Sentiment Analysis Solution
AWS Services Used
The solution uses the following AWS services: AWS S3: Amazon Simple Storage Service (S3) is a scalable object storage service that can store large amounts of data. AWS Lambda: AWS Lambda is a serverless compute service that can run code without provisioning or managing servers. AWS Comprehend: Amazon Comprehend is a natural language processing (NLP) service that can analyze text and extract insights such as sentiment, syntax, entities, and key phrases. AWS SNS: Amazon Simple Notification Service (SNS) is a messaging service that can be used to send notifications to users. AWS Translate: Amazon Translate is a machine translation service that can translate text from one language to another. AWS SQS: Amazon Simple Queue Service (SQS) is a messaging service that can be used to store and process large amounts of messages. AWS CloudWatch: Amazon CloudWatch is a monitoring service that can be used to monitor the solution and generate metrics. AWS Glue: Amazon Glue is a serverless ETL (extract, transform, and load) service that can be used to extract data from the survey feedback file and store it in Amazon S3. AWS Athena: Amazon Athena is a serverless data analytics service that can be used to query and analyze data stored in Amazon S3. AWS QuickSight: Amazon QuickSight is a business intelligence (BI) service that can be used to create visualizations and dashboards based on the sentiment analysis results.
Solution Implementation
The solution implementation is divided into the following steps:
Create an Amazon S3 bucket as raw data bucket to store the survey feedback Excel file.
Uploaded a survey feedback Excel file to the S3 bucket to trigger the AWS Lambda function.
AWS Lambda to extract the survey feedback data from the Excel file, translate the text into English using Amazon Translate, sentiment analysis using Amazon Comprehend, and store the data as Parquet format in another Amazon S3 Bucket.
Create an Amazon SNS topic to notify users by email when the Lambda process data failed.
Create an Amazon CloudWatch to log the lamdba exeuction logs, generate metrics.
AWS Glue Crawler to extract the parquet data from the processed amazon S3 bucket and generate a table schema.
Using Amazon Athena to query the data from the processed Amazon S3 bucket.
Create an Amazon QuickSight dashboard to visualize the sentiment analysis results.
deftranslate_feedbacks(feedbacks): translate = boto3.client('translate') feedbacks_en = [] print(f"calling AWS Translate to auto detect language and translate feedback, total {feedbacks.__len__()}")
for feedback in feedbacks: response = translate.translate_text( Text=feedback, SourceLanguageCode='auto', # Detect source language automatically TargetLanguageCode='en' ) feedbacks_en.append(response['TranslatedText']) print(f"Finished transalate sourceText: {feedback} to targetText: {response['TranslatedText']}")
return feedbacks_en
defcomprehend_sentiment(feedbacks): comprehend = boto3.client('comprehend') batch_size = 25 all_sentiments = [] print(f"calling AWS Comprehend AI to analysis feedback, total {feedbacks.__len__()}")
for i inrange(0, len(feedbacks), batch_size): batch_feedbacks = feedbacks[i:i+batch_size] print(f"calling AWS Comprehend AI to analysis feedback, batch {i} - {i+batch_size}") comprehend_response = comprehend.batch_detect_sentiment(TextList=batch_feedbacks, LanguageCode='en') sentiments = [response['Sentiment'] for response in comprehend_response['ResultList']] all_sentiments.extend(sentiments)
return { 'statusCode': 200, 'body': json.dumps(f'{source_key} File converted to Parquet {target_key} and stored in S3 bucket {target_bucket} successfully') } except Exception as e: print(f'Error processing {source_key}: {e}') sns_client.publish( TopicArn=SNS_TOPIC_ARN, Subject='Lambda Function Processing QE Survey Feedback Failure Notification', Message=f'An error occurred: {str(e)}' )
The above code extracts the survey feedback data from the Excel file, translates the text into English using Amazon Translate, performs sentiment analysis using Amazon Comprehend, and stores the data as Parquet format in another Amazon S3 Bucket. also notify users by email when the Lambda process data failed by using Amazon SNS.
Below is sentiment analysis results visualization using Amazon QuickSight:
AWS Sentiment Analysis Results
This is a high-level overview of the solution implementation. The solution can be further customized and enhanced based on the specific requirements of the app product.
]]>
@@ -267,14 +241,14 @@
+
+
-
-
@@ -282,7 +256,7 @@
https://stonefishy.github.io/2024/09/10/aws-s3-event-replacing-space-with-character-sign-in-object-key-name/2024-09-10T09:22:34.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZThe issue with AWS S3 Event notifications is that it replaces spaces with ‘+’ character sign in the object key name. This can cause issues when trying to access the object in S3. It will occurs NoSuchKey error if not handling this issue properly.
Background
Configured a S3 bucket put event notification to a Lambda function. The Lambda function will be triggered when a new object is uploaded to the S3 bucket. I upload a file named ‘2023 2nd quarter QE survey raw data.xlsx’ into S3 bucket, the Lambda function is triggered, but when I try to access the object in S3, I get NoSuchKey error in AWS CloudWatch Logs. Debugging shows that the object key name is ‘2023+2nd+quarter+QE+survey+raw+data.xlsx’ instead of ‘2023 2nd quarter QE survey raw data.xlsx’. It means the S3 event notification is replacing the space with ‘+’ character sign in the object key name.
The detail error is below:
1
[ERROR] NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
Solution
There are two solutions to fix this issue, one is from AWS S3 upload file side, another is from Lambda function side.
1. From AWS S3 upload file side:
If you have control over the upload process, ensure that the keys are properly URL-encoded.
1 2 3 4 5 6 7 8 9 10 11 12
import boto3 import urllib.parse
s3_client = boto3.client('s3')
# Example key with spaces object_key = "2023 2nd quarter QE survey raw data.xlsx" encoded_key = urllib.parse.quote(object_key)
# Upload the object s3_client.upload_file("/tmp/2023 2nd quarter QE survey raw data.xlsx", "my-bucket-name", encoded_key)
This will ensure that the object key name is properly URL-encoded, which will prevent the S3 event notification from replacing spaces with ‘+’ character sign. The urllib.parse.quote function will replace spaces with ‘%20’ and ‘+’ with ‘%2B’.
2. From Lambda function side:
To fix this issue, we need to modify the Lambda function to handle the object key name with ‘+’ character sign. We can use the urllib.parse package to handle it. The package library provide the function unquote_plus to replace ‘+’ with space. Here is the code to handle the object key name with ‘+’ character sign:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
import os import urllib.parse as urlparse import boto3
s3 = boto3.client('s3')
deflambda_handler(event, context): bucket_name = event['Records'][0]['s3']['bucket']['name'] object_key = event['Records'][0]['s3']['object']['key'] # Replace '+' with space object_key = urlparse.unquote_plus(object_key) # Download the object s3.download_file(bucket_name, object_key, '/tmp/file.txt') # Do something with the downloaded file #...
In the above code, we first get the bucket name and object key from the S3 event notification. We then use the urlparse.unquote_plus function to replace ‘+’ with space in the object key name. Finally, we download the object using the s3.download_file function.
Conclusion
To fix the issue with AWS S3 Event notifications replacing space with ‘+’ character sign in the object key name, we need to handle it properly in the Lambda function. We can use the urllib.parse package to handle it.
在进行复杂的数据比较时，尤其是涉及到 NULL 值时，IS DISTINCT FROM 提供了更直观的比较逻辑。例如，在合并两个数据集时，可以使用此操作符来确保唯一性。
1 2 3 4 5 6 7 8
SELECT * FROM dataset1 FULLOUTERJOIN dataset2 ON dataset1.id ISDISTINCTFROM dataset2.id
这条查询会找出两个数据集中 id 不同的记录，包括 id 为 NULL 的情况。
注意事项
IS DISTINCT FROM 是 SQL 标准中的一部分，但并非所有数据库系统都支持。具体的支持情况需要查阅数据库的文档。在使用 IS DISTINCT FROM 时，确保数据库系统的版本和文档中对此操作符的支持及行为一致。
总结
IS DISTINCT FROM 是一个强大的工具，用于在 SQL 中处理包含 NULL 值的数据比较。它解决了传统比较操作符在处理 NULL 值时的不足，使得数据验证、更新和比较更加准确和可靠。在实际应用中，根据数据库系统的支持情况，合理使用 IS DISTINCT FROM 可以显著提高数据操作的精确性和健壮性。
]]>
+ 在SQL查询中，比较操作符 = 通常用于检查两个值是否相等。然而，当涉及到处理缺失值（NULL）时，这种操作符就会面临挑战。为了解决这一问题，SQL 提供了 `IS DISTINCT FROM` 操作符，它用于精确比较两个值是否不同，即使这些值中有 NULL。本文将详细介绍 IS DISTINCT FROM 的语法、解决的问题以及常见的使用场景。
在进行复杂的数据比较时，尤其是涉及到 NULL 值时，IS DISTINCT FROM 提供了更直观的比较逻辑。例如，在合并两个数据集时，可以使用此操作符来确保唯一性。
1 2 3 4 5 6 7 8
SELECT * FROM dataset1 FULLOUTERJOIN dataset2 ON dataset1.id ISDISTINCTFROM dataset2.id
这条查询会找出两个数据集中 id 不同的记录，包括 id 为 NULL 的情况。
注意事项
IS DISTINCT FROM 是 SQL 标准中的一部分，但并非所有数据库系统都支持。具体的支持情况需要查阅数据库的文档。在使用 IS DISTINCT FROM 时，确保数据库系统的版本和文档中对此操作符的支持及行为一致。
总结
IS DISTINCT FROM 是一个强大的工具，用于在 SQL 中处理包含 NULL 值的数据比较。它解决了传统比较操作符在处理 NULL 值时的不足，使得数据验证、更新和比较更加准确和可靠。在实际应用中，根据数据库系统的支持情况，合理使用 IS DISTINCT FROM 可以显著提高数据操作的精确性和健壮性。
]]>
@@ -342,7 +316,7 @@
https://stonefishy.github.io/2024/08/15/what-is-lag-in-sql/2024-08-15T09:24:21.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357Z最近在做数据分析，需要挖掘数据随时间变化的信息。所有数据物理存储在AWS S3上，通过AWS Glue Catalog和AWS Athena进行数据查询。AWS Athena支持SQL语言，可以对数据进行分析。在处理时间序列数据或分析行间变化时， SQL中的 LAG 函数和 LEAD 函数是非常有用的。下面，我们来看一下 LAG 函数的基本用法。
SELECT month, sales_amount, LAG(sales_amount, 1) OVER (ORDERBYmonth) AS previous_month_sales, sales_amount -LAG(sales_amount, 1) OVER (ORDERBYmonth) AS sales_difference FROM monthly_sales;
LAG 函数是一个强大的工具，可以帮助你在数据分析中处理行间的比较和变化。无论是用于时间序列数据、计算变化量，还是生成滚动报告，LAG 函数都能提供有价值的信息。它通常与 OVER 子句一起使用, 在OVER子句中，我们可以指定分组条件、排序条件等。ORDER BY 确定行的顺序，PARTITION BY 则可以用来将数据分组，使每个分组内的计算互相独立。
]]>
@@ -370,7 +344,7 @@
https://stonefishy.github.io/2024/08/02/how-to-query-tree-structured-relation-data-in-mysql/2024-08-02T10:08:52.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZTo query hierarchical relational data in MySQL, recursive Common Table Expressions (CTEs) are typically used. However, MySQL did not support recursive CTEs before version 8.0, so in earlier versions, self-joins are commonly used to handle such queries. Below is an example using a self-join, assuming we have a table employees that contains information about employees and their manager IDs (manager_id).
Create Table and Insert Data
Creating a table named employees with the following columns:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
CREATETABLE employees ( id INTPRIMARY KEY, name VARCHAR(100), manager_id INT, FOREIGN KEY (manager_id) REFERENCES employees(id) );
We can search for all employees and their direct reports (subordinates) using a self-join. The following SQL statement will list all employees and their direct manager’s name.
1 2 3 4
SELECT e1.name AS employee_name, e2.name AS manager_name FROM employees e1 LEFTJOIN employees e2 ON e1.manager_id = e2.id ORDERBY e1.manager_id, e1.id;
We can also use a self-join to count the number of direct reports for each manager. The following SQL statement will list all managers and the number of their direct reports.
1 2 3 4 5
SELECT e1.name AS manager_name, COUNT(e2.id) AS subordinate_count FROM employees e1 LEFTJOIN employees e2 ON e1.id = e2.manager_id GROUPBY e1.id ORDERBY subordinate_count DESC;
Recursive Common Table Expressions (CTEs)
MySQL 8.0 introduced support for recursive CTEs, which allows us to query hierarchical relational data more efficiently. The following SQL statement will list all employees and their direct reports (subordinates) using a recursive CTE.
1 2 3 4 5 6 7 8 9 10
WITHRECURSIVE subordinates AS ( SELECT id, name, manager_id FROM employees WHERE id =1-- root node CEO, we can replace with any other root node ID, for example 2 which is CTO UNIONALL SELECT e.id, e.name, e.manager_id FROM employees e INNERJOIN subordinates s ON e.manager_id = s.id ) SELECT*FROM subordinates;
But please note that this method only works for MySQL 8.0 and above, as these versions support recursive CTEs.
]]>
@@ -394,7 +368,7 @@
https://stonefishy.github.io/2024/07/26/mysql-8-x-ctes-feature-with-clause/2024-07-26T09:19:59.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZMySQL Common Table Expressions (CTEs) are a powerful feature introduced in MySQL 8.0. CTEs are a type of MySQL 8.0 that provide a way to create temporary result sets that can be referenced within a SELECT, INSERT, UPDATE, or DELETE statement. The primary purpose of `CTEs` is to make complex queries more readable and manageable by breaking them down into simpler.
MYSQL CTEs feature - WITH clause
Purpose of CTEs
Readability: CTEs can make SQL queries more readable, especially for complex queries involving multiple subqueries or recursive operations.
Modularity: They allow you to define a temporary result set that can be reused within the same query, promoting code reuse and reducing redundancy.
Recursive Queries: CTEs support recursive queries, which are useful for querying hierarchical data like organizational charts, bill of materials, or tree structures.
How to Use CTEs
CTEs are defined using the WITH clause and can be referenced within the main query. Here’s the basic syntax:
1 2 3 4 5
WITH cte_name AS ( SELECT ... )
SELECT ... FROM cte_name;
Basic CTE
Suppose you have a table employees and you want to find the average salary of employees in each department.
1 2 3 4 5 6 7
WITH DepartmentSalaries AS ( SELECT department_id, AVG(salary) AS avg_salary FROM employees GROUPBY department_id )
SELECT*FROM DepartmentSalaries;
In this example, DepartmentSalaries is a CTE that calculates the average salary for each department. The main query then selects from this CTE.
CTEs feature also supports multiple temporary result sets in the same query, see below example:
1 2 3 4 5
WITH cte1 AS (SELECT a, b FROM table1), cte2 AS (SELECT c, d FROM table2)
SELECT b, d FROM cte1 JOIN cte2 WHERE cte1.a = cte2.c;
Above sqls defines two CTEs cte1 and cte2 and then joins them using a WHERE clause.
Recursive CTE
A CTE can refer to itself to define a recursive CTE. Common applications of recursive CTEs include series generation and traversal of hierarchical or tree-structured data.
Suppose you have a table employees with a self-referencing column manager_id to represent a hierarchy. You want to find all subordinates of a given manager.
1 2 3 4 5 6 7 8 9 10 11
WITHRECURSIVE Subordinates AS ( SELECT employee_id, manager_id, name FROM employees WHERE manager_id =1-- Starting with the manager ID 1 UNIONALL SELECT e.employee_id, e.manager_id, e.name FROM employees e INNERJOIN Subordinates s ON e.manager_id = s.employee_id )
SELECT*FROM Subordinates;
In this example, Subordinates is a recursive CTE that starts with employees directly reporting to manager ID 1 and then recursively includes all their subordinates.
Key Points
Non-Recursive CTEs: These are straightforward and do not involve recursion. They are simply a way to define a temporary result set for use within the query.
Recursive CTEs: These involve recursion and are useful for hierarchical or tree-structured data. They must include a UNION ALL clause to combine the initial result set with the recursive result set.
Scope: CTEs are scoped to the query they are defined in and cannot be referenced outside of that query.
CTEs are a powerful tool in MySQL that can significantly improve the readability and maintainability of complex SQL queries.
Reference Links:
For other scenarios, like use WITH clause in UPDATE or DELETE statements, please refer to the following links:
By default, the Pulumi import the resource in the region which is specified in the Pulumi.yaml or Pulumi.<stack-name>.yaml file. If we import the resources which is located in other regions. It will cause the error by using pulumi import command.
For example, we have quicksight resources such like DataSource, DataSet located in the eu-west-1 region, we already manage these resources in the pulumi by using pulumi import CLI command. All resources are located in eu-west-1 region. It is specified in the Pulumi.yaml or Pulumi.<stack-name>.yaml file like below.
1 2
config: aws:region:eu-west-1
Now we also want to import the existing resources such like QuickSight user Groups into the pulumi. But the AWS Quicksight user Groups resources all are located in the us-east-1 region. The pulumi will give us the error if we try to import the other region resource direclty.
This is because the Pulumi is using default provider for the AWS resources. The default provider is set to the region which is specified in the Pulumi.yaml or Pulumi.<stack-name>.yaml file. So, if we want to import the resources from other region, we need to specify the provider for that region.
Pulumi Provider
A Pulumi provider is a plugin that enables Pulumi to interact with a specific cloud provider or service. These providers are responsible for translating the Pulumi code into the appropriate API calls for the target cloud platform.
By default, each provider uses its package’s global configuration settings, which are controlled by your stack’s configuration. You can set information such as your cloud provider credentials with environment variables and configuration files. If you store this data in standard locations, Pulumi knows how to retrieve them. For example, you can run below command to set the AWS region to eu-west-1 region for the AWS provider configuration.
1
pulumi config set aws:region eu-west-1
This command actually will set the aws:region configuration value for the AWS provider in your Pulumi stack yaml file. You can also define the provider in your pulumi code, and create related resources in the specified region.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
import pulumi import pulumi_aws as aws
# Create a new provider for the us-east-1 region us_east_1_provider = aws.Provider('us-east-1', region='us-east-1') # Create the Quicksight Groups resources in the us-east-1 region quicksight_group = aws.quicksight.Group( "dev", aws_account_id="<aws-account-id>", group_name="dev", opts=pulumi.ResourceOptions( provider=us_east_1_provider ) )
In above code, we create a new provider for the us-east-1 region and then create the Quicksight user Groups resources in the us-east-1 region. The provider option is used to specify the provider to use for the resource. Even we have global configuration for the eu-west-1 region, we can still create the resources in the us-east-1 region by specifying the provider.
Importing the AWS Resources of the Other Region
Back to previous topic, if we want to import the AWS Quicksight Users and Groups resources from the us-east-1 region in current pulumi stack from the command line, we need to specify the provider for the pulumi command line. The Pulumi CLI import command takes an additional --provider option to specify the provider to use for the import.
1
pulumi import aws:quicksight/group:Group dev xxxxxx/default/dev --provider name=urn
In above command, we are importing the aws:quicksight/group:Group resource with the dev name in the provider. For the --provider option, The name is the name of the provider to use for the import, and urn is the URN of the provider to use for the import. Typically, the resource urns in pulumis is below format.
For the details of Pulumi Resources URNs, please refer to the Pulumi URNs.
In our scenario, we can import the Quicksight Groups resources from the us-east-1 region by using the provider. There is one thing is important to note For example, we don’t have any Provider resources for the us-east-1 region in our current stack. If we run below command to import the Quicksight Groups resources from the us-east-1 region, it will fail. Below is an examle of the full import resource with --provider option
1
pulumi import aws:quicksight/group:Group dev <aws-account-id>/default/dev --provider us_east_1_provider=urn:pulumi:<pulumi-project-name>::quicksight::pulumi:providers:aws::us_east_1_provider
The <aws-account-id> and <pulumi-project-name> are placeholder just for example. Without the Provider resource for the us-east-1 region, the import command will fail as below error message.
1
Preview failed: bad provider reference 'us_east_1_provider=urn:pulumi:<pulumi-project-name>::quicksight::pulumi:providers:aws::us_east_1_provider' is not valid URN'
The error full screenshot is below.
To fix this issue, we need to create the Provider resource for the us-east-1 region in our current stack. We can do this by adding the Provider resource in in our Pulumi code and using Pulumi up command to create the resource.
1 2 3 4 5
import pulumi import pulumi_aws as aws
# Create a new provider for the us-east-1 region us_east_1_provider = aws.Provider('us-east-1', region='us-east-1')
After that, we can run the import command again to import the Quicksight Groups resources from the us-east-1 region. And now you will see the Quicksight Groups resources of the us-east-1 region in your pulumi stack.
Summary
To import the AWS resources of the other region, we need to specify the provider for the pulumi command line. The Pulumi CLI import command takes an additional --provider option to specify the provider to use for the import. The provider resource should be created in pulumi before importing the resources of the other region resource.
]]>
@@ -433,14 +407,14 @@
+
+
-
-
@@ -450,7 +424,7 @@
https://stonefishy.github.io/2024/06/25/keras3-0-a-multi-framework-machine-learning-library/2024-06-25T10:12:36.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZKeras3 is a full rewrite of Keras that enables you to run your Keras workflows on top of either JAX, TensorFlow, or PyTorch, and that unlocks brand new large-scale model training and deployment capabilities. It’s multi-framework machine learning, meaning that you can use Keras to train models on top of different backends, and deploy them to different platforms. You can also use Keras as a low-level cross-framework language to develop custom components such as layers, models, or metrics that can be used in native workflows in JAX, TensorFlow, or PyTorch — with one codebase.
Keras 3 Multi-framework Machine Learning
What’s New in Keras 3?
Keras 3 introduces several exciting features that enhance its usability, performance, and flexibility:
Unified API
Keras 3 continues to build on its legacy of a user-friendly and intuitive API. It aims to unify the high-level and low-level APIs more seamlessly, providing a consistent experience across different backends such as TensorFlow, PyTorch, and others.
Multi-backend Support
While Keras has traditionally been closely associated with TensorFlow, Keras 3 expands its compatibility to other popular deep learning frameworks. This means you can now use Keras with PyTorch and other backends, leveraging Keras’ high-level abstractions and ease of use across different environments.
Improved Performance
Efforts have been made in Keras 3 to optimize performance across various operations, ensuring faster execution times and better utilization of hardware resources. This improvement is crucial for handling larger datasets and complex models efficiently.
Enhanced Model Deployment
Keras 3 simplifies the process of deploying trained models to production environments. With streamlined APIs for model serialization and deployment tools, it becomes easier to integrate Keras models into real-world applications.
Expanded Model Zoo
Keras 3 comes with an expanded model zoo, offering pre-trained models for a wider range of tasks and domains. This includes vision models (e.g., ResNet, EfficientNet), NLP models (e.g., BERT, GPT), and other specialized architectures, all accessible through a unified interface.
Advanced AutoML Capabilities
The new release includes improved AutoML capabilities, allowing developers to automate model selection, hyperparameter tuning, and architecture search. This feature can significantly accelerate the model development process, especially for beginners and researchers exploring new domains.
Pre-trained Models
There’s a wide range of pretrained models that you can start using today with Keras 3. About 40 Keras Applications models (the keras.applications namespace) are available in all backends. These models are pre-trained on large datasets and can be used for transfer learning or fine-tuning. It includes:
Pre-trained Models for Natural Language Processing
Albert
Bart
Bert
Bloom
DebertaV3
DistilBert
Gemma
Electra
Falcon
FNet
GPT2
Llama
Llama3
Mistral
OPT
PaliGemma
Phi3
Roberta
XLMRoberta
Pre-trained Models for Computer Vision
CSPDarkNet
EfficientNetV2
MiT
MobileNetV3
ResNetV1
ResNetV2
VideoSwinB
VideoSwinS
VideoSwinT
VitDet
YOLOV8
ImageClassifier
VideoClassifier
CLIP
RetinaNet
How to Get Started with Keras 3?
1.Install Keras 3
Ensure you have the latest version of Keras installed. You can install Keras via pip if you haven’t already:
1
pip install --upgrade keras
2.Define Model
Use Keras’ high-level API to define your deep learning model. Here’s a simple example of a convolutional neural network (CNN) for image classification:
1 2 3 4 5 6 7 8 9 10 11 12
from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense from keras.models import Sequential
Keras 3 provides a simple and unified interface for deploying trained models to production environments. You can serialize your models and deploy them using tools such as TensorFlow Serving, PyTorch Hub, or JAX Hub.
Summary
Keras 3 bring a lot of exciting features to the table, including multi-backend support, improved performance, and enhanced model deployment. It also includes a wide range of pre-trained models for natural language processing and computer vision, making it easy to get started with deep learning. With these features, Keras 3 is a powerful and flexible tool for building and deploying deep learning models.
]]>
@@ -484,7 +458,7 @@
https://stonefishy.github.io/2024/06/14/understanding-the-x-frame-options-http-header/2024-06-14T14:36:28.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZRecently, we build a frontend website as a nginx docker image, before go live on production. We asking the security team to do the security scan for the website on stage environment. One of security issues indicates the X-Frame-Options HTTP header is not set properly. It will cause the website to be vulnerable to clickjacking attacks.
Clickjacking Attack
Clickjacking is a type of security vulnerability that allows an attacker to trick a user into clicking on a link or button on a malicious website that is designed to look like the legitimate website. This can happen when the attacker embeds the malicious website within a frame on the legitimate website, which can trick the user into clicking on the malicious link or button.
To prevent clickjacking attacks, we can use the X-Frame-Options HTTP header to specify whether a web page can be displayed within a frame or iframe. This header can have three possible values: DENY, SAMEORIGIN, and ALLOW-FROM uri.
What is X-Frame-Options?
The X-Frame-Options is an HTTP response header used to control whether a web page can be displayed within a frame or iframe. It helps to mitigate clickjacking attacks by preventing malicious websites from embedding a vulnerable site within a frame and tricking users into taking unintended actions.
How it works
The X-Frame-Options header can have three possible values: DENY, SAMEORIGIN, and ALLOW-FROM uri.
DENY: This value prevents the page from being displayed in a frame, regardless of the site attempting to do so.
SAMEORIGIN: With this value, the page can be displayed in a frame on the same origin as the page itself. This restricts the frame to the same origin as the parent page.
ALLOW-FROM uri: Here, the page can only be displayed in a frame on the specified origin.
Implementation
To implement the X-Frame-Options header, simply include the header in the server’s HTTP response. It can be implemented on code programming, server configuration, or web server configuration.
Code Programming
Below is an example of how to set the header using different programming languages:
Using Node.js (Express)
1 2 3 4 5
// Set X-Frame-Options header to DENY app.use((req, res, next) => { res.setHeader('X-Frame-Options', 'DENY'); next(); });
Using Django (Python)
1 2
# Set X-Frame-Options header to SAMEORIGIN response['X-Frame-Options'] = 'SAMEORIGIN'
Using ASP.NET (C#)
1 2
// Set X-Frame-Options header to ALLOW-FROM Response.AddHeader("X-Frame-Options", "ALLOW-FROM https://example.com");
Server Configuration
Nginx
To configure Nginx to send the X-Frame-Options header, add this either to your http, server or location configuration:
1
add_header X-Frame-Options SAMEORIGIN always;
Apache
To configure Apache to send the X-Frame-Options header for all pages, add this to your site’s configuration:
1
Header always set X-Frame-Options "DENY"
IIS
To configure IIS to send the X-Frame-Options header for all pages, add this to your web.config file:
To demonstrate the effectiveness of the X-Frame-Options header, we can create a parent html page, and a child html page that is embedded within a frame in the parent page.
Parent HTML Page
Parent HTML page includes the iframe of the child page. The child page is hosted on a different domain (http://localhost:3333/child.html) to demonstrate the effectiveness of the X-Frame-Options header.
1 2 3 4 5 6 7 8 9 10 11
<!DOCTYPE html> <html> <head> <title>Parent Page</title> </head> <body> <h1>Parent Page</h1> <p>This is parent page. below is the iframe of child page.</p> <iframesrc="http://localhost:3333/child.html"frameborder="0"sandbox="allow-scripts"style="width: 100%; height: 200px;"></iframe> </body> </html>
Child HTML Page
Child HTML page is a simple page that displays a message. It is hosted on the domain (http://localhost:3333/child.html) by using httpster tool.
1 2 3 4 5 6 7 8 9 10
<!DOCTYPE html> <html> <head> <title>Child Page</title> </head> <body> <h1>Child Page</h1> <p>This is a child page.</p> </body> </html>
Testing
To test the effectiveness of the X-Frame-Options header, we can open the parent page in a browser and observe the behavior.
Without X-Frame-Options Header
By default, the httpster does not add the X-Frame-Options header to the response. Therefore, the child page can be embedded within a frame on the parent page. See below screenshot, these is no X-Frame-Options header in the response.
Without X-Frame-Options Header
With X-Frame-Options Header
With the X-Frame-Options header set to DENY, the child page cannot be embedded within a frame on the parent page.
To test the X-Frame-Options header, we need to modify the httpster server source code to add the X-Frame-Options header to the response. Actually, the httpster tool is a simple HTTP server base on node express. We can modify the app.use function to set the X-Frame-Options header in the httpster source code.
Here is the modified app.use function with the X-Frame-Options header set to DENY:
We can then open the parent page in a browser and observe the behavior. See below screenshot, these is with X-Frame-Options header value set to DENY in the response. And the child page is blocked from being embedded within a frame on the parent page.
With X-Frame-Options Header Value DENY
And also, there is error message in the console of the browser, which indicates that the child page is blocked from being embedded within a frame on the parent page.
With X-Frame-Options Header Value DENY Console Error
You can also test the X-Frame-Options header with different values such as SAMEORIGIN and ALLOW-FROM uri to see how it affects the behavior of the website.
Conclusion
By implementing the X-Frame-Options header, web developers can enhance the security of their websites and protect users from potential clickjacking attacks. It is recommended to set this header appropriately based on the specific requirements of the web application.
Remember to test the effectiveness of the header using browser developer tools and security testing tools to ensure that it is properly configured.
]]>
@@ -508,7 +482,7 @@
https://stonefishy.github.io/2024/04/18/the-points-of-aws-china-cloudfront-you-need-to-notice/2024-04-18T10:16:54.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZThere are much difference between AWS Global and AWS China. The background of this blog is that I’m responsible for migrating the aws global application to aws china. The application already go lived on AWS Global. The application is collecting the user inforamtion and for business logic. The business wants this application to serve China customer. Due to the application regulation, the application needs to deployed in AWS China and store the user information in AWS China.
The application is using below AWS services:
AWS S3: store the static website assets and user information.
AWS ALB: the load balancer for the application.
AWS ASG: auto scaling group for the application.
AWS ECR: store the application container image.
AWS ECS: run the application container.
AWS ACM: manage the SSL certificate.
AWS WAF: web application firewall.
AWS VPC: virtual private cloud.
AWS S3 VPC Gateway: access the S3 bucket from the VPC.
AWS CloudWatch: monitor the application logs, performance and alarms.
AWS SNS: notificate the stack holder when application performance is abnormal.
AWS CloudFront: serve the static website and user information.
AWS China
The AWS China is a separate entity operated by a local partner in compliance with Chinese regulations. Data centers located in Beijing and Ningxia. The operator is different between Beijing and Ningxia. Beijing region operated by Sinnet(光环新网)，Ningxia region operated by NWCD(西云数据). Basically， the service price of Ningxia region is cheaper than Beijing region. You can find the detail pricing in the AWS China link https://calculator.amazonaws.cn/#/. AWS Fargate priciing is here https://www.amazonaws.cn/en/fargate/pricing
Difference between AWS Global and AWS China
The AWS China has many limiation and difference with AWS Global. And also some new services are not available in AWS China. When you migrate the application to AWS China, you need to consider the below points:
AWS China has different pricing policy. The pricing policy is different between Beijing and Ningxia.
The Infrastructure code is different between AWS Global and AWS China. The code need to be modified to adapt to AWS China.
The Website should be do the ICP filling and Goverment Filling. (域名备案，网安备案)
Infrastructure as Code
We’re using Pulumi to manage the infrastructure as code. Pulumi is a tool for developing, building, and deploying cloud applications and infrastructure. It supports multiple cloud providers including AWS, Azure, GCP, and Kubernetes. There are AWS Service Resource definition is different with AWS Global on AWS China. In AWS China there is an amazonaws.com.cn string for endpoint, and aws-cn ARN prefix. The code need to be modified to adapt to AWS China.
In our application is much difference between AWS Global and AWS China, especially the CloudFront.
Requires ICP filing and domain name filing in AWS China.
The CloudFront provides domain name like “*.cloudfront.cn” which cannot be used in for website serving in AWS China. You can not access the website through the CloudFront domain name. It returns 403 Forbidden error.
The SSL/TLS certificates for CloudFront does not support the Amazon Certificate Manager in AWS China. It requires to use SSL/TLS certificate from third party, and then - - import certificate in IAM. It is only support IAM to store the certificates for CloudFront in AWS China.
The CloudFront does not supports the Amazon WAF in AWS China.
The Cache polices and Origin request polices does not support in AWS China
The Lambda@Edge is not available in AWS China.
CloudFront origin access only supports legacy access identities OAI for S3 bucket, does not support OAC in AWS China
The CloudFront origin for S3 bucket which is not a website endpoint, the following format: bucket-name.s3.region.amazonaws.com.cn, remember region after s3
The CloudFront origin for S3 bucket which is a website endpoint, use the following format: bucket-name.s3-website.region.amazonaws.com.cn, remember region after s3-website
In this blog, we have discussed the important points when migrate the aws global application to aws china, especially for the AWS CloudFront. We have listed the difference between AWS Global and AWS China, and also the CloudFront difference between AWS Global and AWS China.
Recently, we got a requirement from the company to move the application to AWS Cloud. The company has a strong focus on security and compliance, and stack holders also want the application more reliable and scalable. The migration also need to be done as soon as possible.
The application running on a local data center. The application is consists of two parts, frontend is a static website built with React and provide the user interface to user, the backend is a Python Flask application that provide the API to interact with the frontend. The backend server also contains a machine learning model algorithm that is used to process the user’s ears photo.
The application main logic is that the user answer some questions and scan and upload their ears photo to the backend server from the website, the backend server will process the photo and return the suggestion result to user to recommend which headset or earphone is the best fit for them.
Architecture
After analysis application technologies and architecture, base on the requirements, we did some architecture design. Below is the architecture of the application on AWS Cloud.
Application Architecture on AWS Cloud
The application is hosted on AWS Cloud, major is that the frontend is served by CloudFront, the backend is served by ECS, and the user’s ears photo is stored in S3 bucket. The application is using the following AWS services:
AWS S3 Bucket
Setup two s3 buckets, one is for storing the user’s ears photo, and config the object expires after 90 days. second bucket is for storing the static website files. All s3 buckets are public blocked.
AWS VPC
Create a dedicate VPC for the application, and configure the subnets, route tables, and security groups. Two public subnets and two private subnets are used.
AWS ECR
Use ECR to store the Docker image of the backend application. The image will be built and pushed to ECR by CI/CD pipeline.
AWS ECS
Use ECS to run the backend application as a container in private subnets, and configure the auto scaling group and load balancer. Autoscaling minimum size is 2 and maximum size is 20.
AWS ALB
Create a ALB to serve the backend ECS service, and configure the listener rules to forward the traffic to the ECS service. The ALB attached the SSL certificate from ACM.
AWS S3 VPC Gateway Endpoint
Use the S3 VPC Gateway Endpoint to access the s3 bucket from the backend ECS container.
AWS Internet Gateway
The Internet Gateway to connect the VPC to the internet. Put the ALB on the two public subnets across two AZs
AWS CloudWatch
Use CloudWatch to monitor the application performance, and create alarms to notify the team when the application is not running as expected.
AWS SNS
Use SNS to notify the team when the application performance is not good, and the team can take action to improve the application performance.
AWS ACM
Use ACM to manage the SSL certificate for the ALB and CloudFront, the certificate is issued by the IT team. The application is served over HTTPS.
AWS CloudFront
Use CloudFront to serve the static website files, and cache the files to improve the website loading speed. Config CloudFront to access s3 bucket by OAC. Create a another origin for the ALB.
AWS Security Group
Create a security group for the ECS container, and allow the traffic from the ALB to the ECS container. And one more security group for the ALB to allow the traffic only from AWS CloudFront prefix list.
AWS IAM
Create an IAM role for the ECS container, and attach the necessary policies to the role to access the s3 bucket, ECR, and CloudWatch.
AWS WAF
Use WAF to protect the application from common web exploits and attacks. This is mandatory for the company’s security policy. The security team will also review the infrastucture and do the security scan the application. The application won’t be deployed to production if the security scan failed.
IaC with Pulumi
Use Pulumi to manage the AWS resources, and create the infrastructure as code. The code will be checked into the source control, and for the pipeline, we’re using Bamboo pipeline as company already using Bamboo for CI/CD. The pipeline will doing below major things.
Build the Docker image and push to ECR.
Deploy the frontend static website to CloudFront, and invalidate the cache to make the content updated for end user.
Update the infrastucture by creating or updating the AWS resources by using pulumi.
Rationale
The migration of the legacy application to AWS Cloud is a complex task, and we need to follow the best practices to make the migration successful.
Using CloudFront and S3 bucket to host the static website and user’s ears photo is scalable and cost-effective.
Using the ECS and ALB to serve the backend application is also a good choice to improve the application performance and scalability. We’re not using AWS API Gateway and AWS Lambda to serve as backend because we are requested to migrate the application to Cloud as soon as possible. Build the python Flask application to a docker image and push to ECR is a good practice to deploy the application to AWS Cloud in this situation.
Using the VPC and security group to isolate the application and improve the security is a must. The ECS is located in private subnets, and the ALB is in public subnets, and the traffic is only allowed from AWS CloudFront prefix list to ALB, then forward traffic to ECS container.
Using the ACM to manage the SSL certificate for the ALB and CloudFront is a good practice to improve the security and compliance.
Using the CloudWatch to monitor the application performance and create alarms to notify the team when the application is not running as expected is a good practice to improve the application reliability.
Using the IAM role to access the s3 bucket, ECR, and CloudWatch is a good practice to improve the security and control.
Using the WAF to protect the application from common web exploits and attacks is a mandatory requirement for the company’s security policy.
Using Pulumi to manage the AWS resources as code is a good practice to improve the automation and reliability of the migration process.
Summary
This is just a sample of how to migrate a legacy application to AWS Cloud, and there are many other factors to consider when migrating a legacy application to AWS Cloud. The key is to follow the best practices and use the right tools to make the migration successful base on the requirements.
]]>
@@ -549,14 +523,14 @@
+
+
-
-
@@ -568,7 +542,7 @@
https://stonefishy.github.io/2024/02/27/importing-existing-cloud-resources-with-pulumi/2024-02-27T14:26:24.000Z
- 2024-11-15T06:39:09.206Z
+ 2024-11-18T09:56:32.357ZIn many real-world scenarios, cloud infrastructure is already in place before adopting infrastructure as code (IaC) solutions like Pulumi. Pulumi provides a feature called import to help manage existing cloud resources within its IaC framework. This feature allows users to import the current state of resources into their Pulumi codebase, making it easier to adopt Pulumi for managing existing infrastructure.
Pulumi Import
Pulumi’s import feature provides a way to bring existing cloud resources under Pulumi’s management. By creating a Pulumi program and using the pulumi import command, users can declare and manage existing infrastructure resources using Pulumi. The pulumi supports both importing existing resources with the CLI and importing existing resources in the code. Here we’re talking about the CLI import to generate the code for the imported resources.
Usage and Syntax
To import an existing cloud resource into Pulumi, you need to follow these steps:
Create a Pulumi Project Create a new Pulumi project or use an existing Pulumi project where you want to manage the imported resources. For creating a pulumi project, you can check the previous blog post on how to create a new Pulumi project.
Identify the Resource to Import Identify the existing resource in your cloud provider environment that you want to import into Pulumi. This could be a virtual machine, database, storage bucket, or any other supported resource.
Apply the Import Apply the import operation to bring the existing resource under Pulumi’s management. Pulumi will generate the appropriate code for the resource based on its current state in the cloud provider environment.
The syntax for the import command is as follows:
1
pulumi import <type> <name> <id>
<type> is the Pulumi type token to use for the imported resource.
<name> is the resource name to apply to the resource once it’s imported.
<id> is the value to use for the resource lookup in the cloud provider.
Managing Imported Resources
Once the resources are imported, they can be managed just like any other Pulumi-managed resources. The imported resources can be updated, deleted, and included in stacks alongside other Pulumi-declared infrastructure.
Example
I created a S3 bucket name my-s3-bucket from AWS Console manually. But now I want to manage this S3 bucket by Pulumi. After identifying the bucket to be imported, the import command:
aws:s3/bucket:Bucket is the Pulumi type token for the S3 bucket resource.
my-bucket is the resource name to apply to the imported resource.
my-s3-bucket is the value to use for the resource lookup in the AWS provider, here it’s bucket name.
After running the import command, Pulumi will generate the appropriate code for the S3 bucket resource based on its current state in the AWS provider. Below is screenshot of the output of the import command:
In above code, you will notice there is a protect=True option set for the imported resource. This is to prevent any accidental deletion of the imported resource.
So when you try to delete the imported resource, Pulumi will give the errors to you. Let’s try to delete the imported S3 bucket:
1
pulumi destroy
You see, it displays the error message that the S3 bucket is protected and cannot be deleted.
Pulumi Destory Import Protected Resource
If you want to delete the resource in the cloud provider environment, you can remove the protect=True option from the code or change the protect option to False.
In above we’re using pulumi import to import the s3 bucket resource and code is generated on console. We can also generate the code into python file directly by using below command:
Pulumi maintains a state file that tracks the current state of all resources in the cloud provider environment. When a resource is imported, Pulumi updates the state file to reflect the imported resource. This allows Pulumi to manage the imported resource as if it were created in the cloud provider environment.
Sometimes, we want to delete the state which imported in pulumi, but keep the existing cloud resources. In such case, we can use below command to only delete the state and keep the existing cloud resources.
1
pulumi state delete <urn>
<urn> is the unique resource identifier of the imported resource.
To check the <urn> of the resource, we can use pulumi stack --show-urns to see the list urns of all resources in the current stack.
1
pulumi stack --show-urns
Pulumi Stack Show Urns
In above screenshot, we can see the <urn> of the imported S3 bucket resource.To delete the state of the imported resource, we can use the following command:
1
pulumi state delete urn:pulumi:dev::pulumi-test::aws:s3/bucket:Bucket::my-bucket --force -y
Pulumi State Delete Imported Resource
After deleting the state, the imported S3 bucket will still exist in the cloud provider environment.
Conclusion
Pulumi’s import feature allows users to seamlessly integrate existing cloud resources into their Pulumi programs. By following the import process and syntax, users can effectively manage their entire infrastructure, including existing resources, through Pulumi’s IaC approach.
This feature simplifies the transition to Pulumi for managing infrastructure and enables teams to leverage the benefits of IaC without having to recreate their entire cloud environment from scratch.
]]>
@@ -583,14 +557,46 @@
+
+
+
+
+
+
+
+ Pulumi - A Powerful IaC to manage the cloud infrastructure
+
+ https://stonefishy.github.io/2024/02/11/pulumi-a-powerful-iac-to-manage-the-cloud-infrastructure/
+ 2024-02-11T15:12:13.000Z
+ 2024-11-18T09:56:32.357Z
+
+ To manage the application cloud infrastructure more efficiently, we can use the Terraform for IaC(Infrastructure as Code). But today, we’re not going to talk about the Terraform, we’re going to talk about the Pulumi. A powerful IaC tool that manages the cloud infrastructure.
Pulumi Platform
Pulumi is an open-source infrastructure as code (IaC) tool that provides a powerful way to create, deploy, and manage cloud infrastructure. It is the easiest way to build and deploy infrastructure, of any architecture and on any cloud, using programming languages that you already know and love, such as TypeScript, Python, Go, C#, Java etc.
It is a cross-platform tool that runs on Windows, Linux, and macOS, and supports a wide range of cloud providers, including AWS, Azure, GCP, Kubernetes, Docker, and more. It is also easy to use and has a simple and intuitive interface.
CI/CD integration is also supported, which means you can use Pulumi to deploy your infrastructure as part of your CI/CD pipeline. This makes it easier to manage and update your infrastructure as your application evolves.
Install Pulumi
The pulumi is a cross-platform tool that runs on Windows, Linux, and macOS. You can find and download the latest version from the official website: https://www.pulumi.com/docs/install/versions/. Follow this link to https://www.pulumi.com/docs/install/ to install the Pulumi CLI. It’s very simple to set up the Pulumi CLI on your machine.
Once you installed pulumi, simply run the below command to check the version:
1
pulumi version
Create a new Pulumi project
To create a new Pulumi project, you can use the pulumi new command. Below command is creating a new project with AWS Python template.
1
pulumi new aws-python
This will create a new project with a simple AWS Python template. The project will have a Pulumi.yaml file, which is the configuration file for the project.
Configure the Pulumi project
The Pulumi.yaml file is the configuration file for the project. It contains the project name and some configuration.
In the above configuration, we have set the project name as my-project, the runtime as python and the virtualenv as venv. The description is a brief description of the project.
Create a new stack
To create a new stack, you can use the pulumi stack init command. Below command is creating a new stack with the name dev.
1
pulumi stack init dev
Once you created the stack, the pululmi will generate a file named Pulumi.dev.yaml in your project folder. You can select it using the pulumi stack select command.
1
pulumi stack select dev
Configure the stack
The Pulumi.dev.yaml file is the configuration file for the stack.
In above configuration, we have set the encryption salt (this is generated), and the AWS region and profile. You can add more configuration as per your requirement.
Create a new resource
To create a new resource, such as s3 bucket, you can write python code in the main.py file. Below is the code to create a new s3 bucket.
1 2 3 4
import pulumi import pulumi_aws as aws
bucket = aws.s3.Bucket("my-bucket")
In the above code, we have imported the aws module and created a new s3 bucket resource. The Bucket function creates a new s3 bucket with the name my-bucket.
Preview the changes
To preview the changes, you can use the pulumi preview command. This command will show the changes that will be applied to the infrastructure.
1
pulumi preview
Below is the project currently I’m working on for pululmi preview showcase.
Pulumi Preview
In above screenshot, you can see the changes that will be applied to the infrastructure. Including update, create and delete resources listed. You can also use the --diff option to show the difference between the current state and the desired state.
1
pulumi preview --diff
Deploy the infrastructure
To deploy the infrastructure, you can use the pulumi up command. This command will deploy the infrastructure as per the configuration in the Pulumi.yaml file.
1
pulumi up
This will deploy the infrastructure and show the output.
Check the status of the infrastructure
To check the status of the infrastructure, you can use the pulumi stack command. This command will show the status of the infrastructure.
1
pulumi stack
This will show the status of the infrastructure. Below is the output of the pulumi stack command of one project I’m working.
Pulumi Stack
Destroy the infrastructure
To destroy the infrastructure, you can use the pulumi destroy command. This command will destroy the infrastructure as per the configuration in the Pulumi.yaml file.
1
pulumi destroy
This will destroy the infrastructure. Please be aware that this command will destroy all the resources in the stack. It’s dangerous to use this command, so use it with caution. You should know what you’re doing before using this command.
There are much more features of Pulumi, but I hope this article will give you a good idea about Pulumi.
Conclusion
Pulumi is a powerful IaC tool that manages the cloud infrastructure. It is easy to use and has a simple and intuitive interface. It supports a wide range of cloud providers, including AWS, Azure, GCP, Kubernetes, Docker, and more. It is also easy to integrate with CI/CD pipeline.
]]>
+
+
+
+
+ <p>To manage the application cloud infrastructure more efficiently, we can use the <code>Terraform</code> for <code>IaC(Infrastructure as Co
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/categories/Data-Analysis/index.html b/categories/Data-Analysis/index.html
index 265c958b..8eb61a8d 100644
--- a/categories/Data-Analysis/index.html
+++ b/categories/Data-Analysis/index.html
@@ -537,17 +537,17 @@
Previously, we created AOP with Spring framework via implementing interface of spring AOP. And now we can use XML Schema to achieve it. We need to add below AspectJ libraries to our project. So downloading and adding them to the CLASSPATH of application, and also we can use the maven to build our application.
Previously, we created AOP with Spring framework via implementing interface of spring AOP. And now we can use XML Schema to achieve it. We need to add below AspectJ libraries to our project. So downloading and adding them to the CLASSPATH of application, and also we can use the maven to build our application.
What is AOP?, AOP stand for Aspect Oriented Programming. It’s very important module for Spring framework. In the enterprise level application programming we used to add different types of services to our application at runtime automatically, like logging, email, transaction, authentication services. These actions we called cross-cutting functionalities. AOP do this job.
What is AOP?, AOP stand for Aspect Oriented Programming. It’s very important module for Spring framework. In the enterprise level application programming we used to add different types of services to our application at runtime automatically, like logging, email, transaction, authentication services. These actions we called cross-cutting functionalities. AOP do this job.
HTML5 adds new common attributes for original HTML Tag, and these attributes enhances the HTML elements’s function.
-
contentEditable
The contentEditable attribute supports most tags. The browser allows user edits the content of element if this attribute is set to true. These elements is not like input or textarea tags. They are not support editable content like table, div, span and so on these tags.
HTML5 adds new common attributes for original HTML Tag, and these attributes enhances the HTML elements’s function.
+
contentEditable
The contentEditable attribute supports most tags. The browser allows user edits the content of element if this attribute is set to true. These elements is not like input or textarea tags. They are not support editable content like table, div, span and so on these tags.
Ractive.js is a template-driven UI library, it transforms your templates into blueprints for application that are interative by default. Ractive.js is very similar with AngularJS, and it supports Two-way binding, animations as well. Another powerfull feature is SVG support.
-
Template + Data = UI
You just need to prepare your html template and datas, and the Ractive.js will automatically to combine them then give you what you want. Whe you change your datas, it intelligently updates the real DOM.
-
Usage
First, you need to include Ractive.js to your application. Just like include other javascript librarys easily. You can download it to local and then include it. And if you want include latest version, you just put below codes to your project.
Ractive.js is a template-driven UI library, it transforms your templates into blueprints for application that are interative by default. Ractive.js is very similar with AngularJS, and it supports Two-way binding, animations as well. Another powerfull feature is SVG support.
+
Template + Data = UI
You just need to prepare your html template and datas, and the Ractive.js will automatically to combine them then give you what you want. Whe you change your datas, it intelligently updates the real DOM.
+
Usage
First, you need to include Ractive.js to your application. Just like include other javascript librarys easily. You can download it to local and then include it. And if you want include latest version, you just put below codes to your project.
The reset command moves the current branch to another position, and optionally updates the stage and the working directory. It also is used to copy files from the history to the stage without touching the working directory.
-
If a commit is given with no filenames, the current branch is moved to that commit, and then the stage is updated to match this commit. If –hard is given, the working directory is also updated. If –soft is given, neither is updated.
The reset command moves the current branch to another position, and optionally updates the stage and the working directory. It also is used to copy files from the history to the stage without touching the working directory.
+
If a commit is given with no filenames, the current branch is moved to that commit, and then the stage is updated to match this commit. If –hard is given, the working directory is also updated. If –soft is given, neither is updated.
+ 
+ 
+ 
+ 
- 
- 
- 
+ 
+ 
- 
