Merge pull request #266 from PretendoNetwork/dev

blogposts/12-23-23.md

@@ -0,0 +1,73 @@
+---
+title: "Nintendo Network shutdown - The beginning of the end"
+author: "Jon"
+author_image: "https://www.github.com/jonbarrow.png"
+date: "December 23, 2023"
+caption: "Information regarding Nintendo's rollout of the Nintendo Network shutdown"
+cover_image: "/assets/images/blogposts/12-23-23.jpg"
+---
+
+## EDIT December 27th, 2023: THIS INFORMATION IS NOW OUTDATED. SEE https://pretendo.network/blog/12-27-23. THIS POST WILL REMAIN UP ONLY FOR HISTORICAL REASONS
+
+## Intro
+First, we would like to apologize for the lack of blog posts this year. We planned to do more, but other priorities kept getting in the way. We continued to provide updates through our Discord and social medias during this time, but we plan to use this blog more often now.
+
+This blog post will be a bit more serious than previous posts, as the subject matter is rather somber. We apologize for the lack of energy and quips you may have enjoyed in previous posts. This post will have information regarding several aspects of the shutdown, some of which have been covered on other social media posts. Please use the table of contents below to jump to your desired section.
+
+## Table of Contents
+1. [The Shutdown](#the-shutdown)
+2. [Super Mario Maker](#super-mario-maker)
+3. [New Accounts (Prerequisite)](#new-accounts-prerequisite)
+4. [New Accounts](#new-accounts)
+5. [Conclusion](#conclusion)
+
+## The Shutdown
+In October of 2023, Nintendo [announced the shutdown of Nintendo Network](https://en-americas-support.nintendo.com/app/answers/detail/a_id/63227/~/announcement-of-discontinuation-of-online-services-for-nintendo-3ds-and-wii-u) for April of 2024. It was stated that a specific date and time would be announced later, and as of this post that has not happened yet. There have been no public announcements of any services shutting down early, though Nintendo stated they reserve they right to do so if:
+
+> *"an event occurs that would make it difficult to continue online services for Nintendo 3DS and Wii U software"*
+
+However, it appears that Nintendo has begun a slow rollout of shutdowns. Intentional or not.
+
+## Super Mario Maker
+This information was also detailed in a [Twitter thread](https://twitter.com/PretendoNetwork/status/1736325668412031255). You may read this there if you wish, or read the repost below.
+
+Some time in early December 2023, Super Mario Maker began throwing error `106-0502` very often when trying to connect. It is unclear when this truly began, but the earliest occurrences we could find were around the 8th of December. Upon inspection, the cause for this error was clear: the server the game was attempting to connect to no longer existed. Our first assumption was that this was the beginning of the shutdown early, that Nintendo may have started turning off games without notice. After some more research, however, we discovered some users able to still connect.
+
+Games on Nintendo Network function using 3 servers:
+
+1. The account server, which tells the client the location of the game's authentication server and provides the client with a token to connect with.
+2. The game's authentication server, which logs the user in using the internal NEX account (details on this in the [New Accounts (Prerequisite)](#new-accounts-prerequisite) section), and tells the client the location of the game's secure server.
+3. The game's secure server, which is where all the game play content happens at.
+
+Every game on Nintendo Network uses the same 2 authentication servers. However each game has many dedicated secure servers. When logging in, the authentication server gives one of many randomly selected addresses to connect to. We believe this is some form of load balancing, splitting connections between many servers. The issue resulting in `106-0502` is that the authentication server is now giving clients addresses to dead servers. So far, we have only been able to identify a single working address for the game.
+
+We believe this is due to Nintendo attempting to scale back how many servers are running for each game, to save on costs as the shutdown date approaches. They most likely made an error in their presumed load balancer to not remove these now dead servers from the pool of available addresses. We believe this was done completely unintentionally, however Nintendo shows no signs of fixing this error.
+
+## New Accounts (Prerequisite)
+As briefly touched on in [Super Mario Maker](#super-mario-maker), Nintendo actually uses a series of internal accounts for several services. For games, these are called "NEX accounts". Some background knowledge is required for this section, which will be gone over now.
+
+NEX is the software Nintendo uses for all 3DS and WiiU games (and some Switch games). It is based on a library called Rendez-Vous, made by Canadian software company [Quazal](https://web.archive.org/web/20040610104624/http://www.quazal.com/modules.php?op=modload&name=Sections&file=index&req=viewarticle&artid=101&page=1). Quazal, before being bought out by Ubisoft, would license Rendez-Vous to anyone. Many games, on many platforms, by many developers, all use some variation of Rendez-Vous as it was highly extensible. This is why our server libraries are [theoretically compatible with Ubisoft games](https://twitter.com/PretendoNetwork/status/1727016210435641508).
+
+Nintendo licensed Rendez-Vous and modified it quite a bit, stripping out some more complex features and adding in some custom protocols, rebranding it to NEX. The important takeaway here is that Nintendo did not build this system from scratch, and instead modified a system made for generic game server usage.
+
+## New Accounts
+Some time in late December 2023, new accounts could no longer go online in any games on both the WiiU and 3DS.
+
+As mentioned in [New Accounts (Prerequisite)](#new-accounts-prerequisite), Nintendo did not make their game server software from scratch. Rendez-Vous comes with its own account system, likely due to it being designed for use in one-off games. At the time of its development, online multiplayer games were still very new and the concept of a unified account system (such as Nintendo Network or Uplay) spanning many different games made by the same company, was not a thing. Therefore, Nintendo needed to cope with this existing account system in some way.
+
+On the 3DS, the console registers a new NEX account on the Friends server once during the console's setup process. Each 3DS, under normal circumstances, will only ever have a single account at a time (though this is not *always* true, but not relevant). This is why a NNID is not required to play most games online on the 3DS, as games use your NEX account to login to all game servers. Because of this, a new NEX account will never be created again unless the old one is removed using CFW or by factory resetting the console.
+
+On the WiiU, a new NEX account is created automatically when the NNID is registered, and the NEX account is linked to this NNID. The console then alerts the Friends server of the new NEX account, much like 3DS. Despite using NNIDs, the WiiU also uses these NEX accounts to go online in all games.
+
+Once a NEX account has been created, the Friends server synchronizes this account with all other game servers on Nintendo Network. This is due to the aforementioned account system provided by Rendez-Vous. Due to each game having their own server, and Rendez-Vous not having the concept of a unified account system, each game server has its own account system internally, which is synchronized with other games. It is actually possible to create accounts in other servers besides the Friends server if using a modified client, and those accounts will be usable in other games.
+
+This synchronization process has been stopped. New NEX accounts are no longer being synchronized with any other servers on the network. This affects all games. Because of this, any attempt to connect to any game using an account made after this change was made will result in error `106-0303` on the WiiU and `006-0303` on the 3DS, internally called `RendezVous::InvalidUsername`.
+
+It is not clear at this time whether this was intentional or not.
+
+## Conclusion
+We ask that you do not spam Nintendo's support lines with these issues. Doing so may cause Nintendo to pull the plug entirely early. There are only 3 possible scenarios:
+
+1. Nintendo is aware of these issues already, and spam would annoy them to the point of pulling the plug.
+2. Nintendo is not aware of these issues, and alerting them may cause them to pull the plug. They have already stated that anything making things "difficult" may come with an early shutdown.
+3. These issues were intentional, and spam would annoy them to the point of pulling the plug.

blogposts/12-27-23.md

@@ -0,0 +1,56 @@
+---
+title: "Fortunate updates with Nintendo Network"
+author: "Jon"
+author_image: "https://www.github.com/jonbarrow.png"
+date: "December 27, 2023"
+caption: "Updates regarding our last blog post"
+cover_image: "/assets/images/blogposts/12-23-23.jpg"
+---
+
+### *Edit December 27th, 2023 7:35 PM UTC: Nintendo has fixed the friends sync issue during maintenance*
+
+## Intro
+This is an update to our [last blog post](https://pretendo.network/blog/12-23-23), where we discussed some issues regarding Nintendo Network and how they could possibly be linked to the upcoming shutdown. This post aims to provide updates on the situation, as things have begun to improve.
+
+This is a developing situation. As such, this blog post may be updated at any time as new information comes in. Edits will be mentioned above this introduction.
+
+## For the media
+We deeply appreciate members of the media wanting to cover this situation, and using us as a trusted source for information. We saw a number of outlets covering our last blog post, and we think it's wonderful they are trying to spread the word. With that said, some outlets did a less than stellar job at reporting our last post, not covering some topics fully or accurately. We would like to ask that members of the media who wish to cover anything posted by us please contact us first to verify the accuracy and legitimacy of the information. We have contacts open on most major social media, and would be happy to discuss things.
+
+## Table of Contents
+1. [Thanks](#thanks)
+2. [New Account Syncing](#new-account-syncing)
+2. [New Account Syncing (Friends)](#new-account-syncing-friends)
+3. [Super Mario Maker (Dead Servers)](#super-mario-maker-dead-servers)
+4. [Super Mario Maker (AWS)](#super-mario-maker)
+5. [Conclusion](#conclusion)
+
+## Thanks
+Before we begin, I'd like to thank members of both our community and Nintendo for their parts in this situation. In our last blog post we urged users to not spam Nintendo regarding these issues, which many followed. We would like to thank those who followed that advice, as we have been given confirmation that those actions may have led Nintendo to act "in a way that's negative". We were told this by our media contact, who will remain anonymous. This contact was able to reach out to Nintendo directly, and together we were able to privately forward these issues to the correct channels.
+
+We noted in our last blog post that it was unclear whether these issues were intentional or not, and it appears that this was indeed unintentional. While we cannot confirm whether or not Nintendo will take action on all issues presented to them, we would like to thank Nintendo for taking the time to at least acknowledge the issues and look into correcting them. We would also like to specifically thank the engineers and other IT staff at Nintendo working to maintain these legacy servers despite the impending shutdown.
+
+## New Account Syncing
+In our last blog post we went into detail about how some parts of Nintendo's server architecture works, and how this led to new users on both platforms not being able to connect to game servers anymore. We are happy to announce that as of December 26th, 2023 at around 5:00 PM UTC this seems to have been corrected. We have verified this with several people internally, on both platforms and in several games. New accounts may now go online as normal.
+
+## New Account Syncing (Friends)
+Shortly after our last blog post, we discovered that new friends on the 3DS were also not being synced to other servers even for existing users. This meant that games such as Animal Crossing: New Leaf could not be played with 3DS friends made after the recent server changes, as your friends were no longer being synced. This issue has not yet been confirmed to happen on the Wii U, however due to the Wii U and 3DS sharing a Friends server it is likely this happens on both platforms. We did not publicly disclose this at the time as we were still gathering information in order to accurately report on the issue. Following maintenance on December 27th, 2023 this has been fixed.
+
+## Super Mario Maker (Dead Servers)
+Previously announced via Twitter, and briefly touched on in our last blog post, Super Mario Maker's authentication server has an error with the list of game servers it gives to the client to connect to. Most of these servers are no longer online, meaning most attempts to connect to the game will fail. As of writing on December 27th, 2023, this has not yet been fixed. The issue has been raised to Nintendo.
+
+## Super Mario Maker (AWS)
+For several months we have been aware of a previously undisclosed issue with Super Mario Maker regarding their use of AWS, specifically S3. Super Mario Maker uploads content such as courses and maker profiles to S3 using a NEX protocol called `DataStore`. This protocol gives the client a way to interact with, and upload new, S3 objects. S3 is a service, originally created by Amazon, to manage objects in a secure, scalable, way. For more information on S3 see the official [AWS docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html).
+
+When a client connects to Super Mario Maker for the first time, it first checks if your account has a "maker" object in S3 using `DataStore`. If not, the server gives the client a URL it can use to upload your "maker" object. Think of this as simply a file with some metadata that will describe you and your stats as you play the game. This object is required for continuing online, and will not function without it.
+
+On June 28th, 2022, AWS announced that they will be dropping support for clients using TLS versions 1.0 and 1.1 in [a blog post](https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints/), making TLS 1.2 the minimum required version. These changes take effect in December of this year. The details of what TLS is is not important, but for more details [see here](https://www.cloudflare.com/learning/ssl/transport-layer-security-tls/).
+
+The Wii U and 3DS do not support TLS versions 1.2 or higher, only supporting TLS versions 1.0 and 1.1. Because of the announced changes to AWS, any client using an out of date TLS version would no longer be able to connect to their services. Nintendo seems to have been aware of these TLS changes, as all games which use the `DataStore` protocol were updated at some point to no longer connect directly to AWS. They are now instead proxied through a Nintendo-owned server. We believe this to be simply a TLS proxy.
+
+The exception to this being Super Mario Maker. Since its release it has always connected directly to AWS, and was not updated with the rest of the games to use this Nintendo-owned server. Because of this, Super Mario Maker was the only game vulnerable to these AWS changes. This would prevent the game from uploading the "maker" object for any new users, rendering them unable to play online.
+
+As of December 26th, 2023, Super Mario Maker's game server has been updated to now use this Nintendo-owned proxy. The changes made to AWS will no longer affect the game.
+
+## Conclusion
+We would again like to thank our community members and Nintendo for their roles in all of this. Sorting these issues out is a complex task, and we thank our community members who have not spam contacted Nintendo's support hotlines regarding these issues. The customer service reps on these hotlines likely have no influence over the day to day operations of these servers. Additionally, the issues mentioned here have already been brought to the proper channels. We will continue to update this blog post as new information comes in.