[feature] enable + document explicit IP dialer allowing/denying (#1950)

* [feature] enable + document explicit IP dialer allowing/denying

* lord have mercy

* allee jonge

* shortcut check ipv6 prefixes

* comment

* separate httpclient_test, export Sanitizer

cmd/gotosocial/action/server/server.go

@@ -105,8 +105,12 @@ var Start action.GTSAction = func(ctx context.Context) error {
 	// Set the state storage driver
 	state.Storage = storage
 
-	// Build HTTP client (TODO: add configurables here)
-	client := httpclient.New(httpclient.Config{})
+	// Build HTTP client
+	client := httpclient.New(httpclient.Config{
+		AllowRanges: config.MustParseIPPrefixes(config.GetHTTPClientAllowIPs()),
+		BlockRanges: config.MustParseIPPrefixes(config.GetHTTPClientBlockIPs()),
+		Timeout:     config.GetHTTPClientTimeout(),
+	})
 
 	// Initialize workers.
 	state.Workers.Start()

docs/configuration/httpclient.md

@@ -0,0 +1,56 @@
+# HTTP Client
+
+## Settings
+
+```yaml
+################################
+##### HTTP CLIENT SETTINGS #####
+################################
+
+# Settings for OUTGOING http client connections used by GoToSocial to make
+# requests to remote resources (status GETs, media GETs, inbox POSTs, etc).
+
+http-client:
+
+  # Duration. Timeout to use for outgoing HTTP requests. If the timeout
+  # is exceeded, the connection to the remote server will be dropped.
+  # A value of 0s indicates no timeout: this is not advised!
+  # Examples: ["5s", "10s", "0s"]
+  # Default: "10s"
+  timeout: "10s"
+
+  ########################################
+  #### RESERVED IP RANGE EXCEPTIONS ######
+  ########################################
+  #
+  # Explicitly allow or block outgoing dialing within the provided IPv4/v6 CIDR ranges.
+  #
+  # By default, as a basic security precaution, GoToSocial blocks outgoing dialing within most "special-purpose"
+  # IP ranges. However, it may be desirable for admins with more exotic setups (proxies, funky NAT, etc) to
+  # explicitly override one or more of these otherwise blocked ranges.
+  #
+  # Each of the below allow/block config options accepts an array of IPv4 and/or IPv6 CIDR strings.
+  # For example, to override the hardcoded block of IPv4 and IPv6 dialing to localhost, set:
+  #
+  #   allow-ips: ["127.0.0.1/32", "::1/128"].
+  #
+  # You can also use YAML multi-line arrays to define these, but be diligent with indentation.
+  #
+  # When dialing, GoToSocial will first check if the destination falls within explicitly allowed IP ranges,
+  # then explicitly blocked IP ranges, then the default (hardcoded) blocked IP ranges, returning OK on the
+  # first allowed match, not OK on the first blocked match, or just defaulting to OK if nothing is matched.
+  #
+  # As with all security settings, it is better to start too restrictive and then ease off depending on
+  # your use case, than to start too permissive and try to close the stable door after the horse has
+  # already bolted. With this in mind:
+  # - Don't touch these settings unless you have a good reason to, and only if you know what you're doing.
+  # - When adding explicitly allowed exceptions, use the narrowest possible CIDR for your use case.
+  #
+  # For reserved / special ranges, see:
+  # - https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
+  # - https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
+  #
+  # Both allow-ips and block-ips default to an empty array.
+  allow-ips: []
+  block-ips: []
+```

docs/faq.md

@@ -1,29 +1,50 @@
 # Frequently Asked Questions
 
-- **Where's the user interface?** GoToSocial is just a bare server for the most part and is designed to be used thru external applications. [Semaphore](https://semaphore.social/) and [Tusky](https://tusky.app/) are the best-supported, but anything that supports the Mastodon API should work, other than the features GoToSocial doesn't yet have. Permalinks and profile pages are served directly thru GoToSocial as well as the settings panel, but most interaction goes thru the apps.
+## Where's the user interface?
 
-- **Why aren't my posts showing up on my profile page?** Unlike Mastodon, the default post visibility is Unlisted. If you want something to be visible on your profile page, the post must have Public visibility.
+GoToSocial is just a bare server for the most part and is designed to be used thru external applications. [Semaphore](https://semaphore.social/) and [Tusky](https://tusky.app/) are the best-supported, but anything that supports the Mastodon API should work, other than the features GoToSocial doesn't yet have. Permalinks and profile pages are served directly through GoToSocial as well as the settings panel, but most interaction goes through the apps.
 
-- **Why aren't my posts showing up on other servers?** First check the visibility as noted above. TODO: explain how to debug common federation issues
+## Why aren't my posts showing up on my profile page?
 
-- **Why am I getting frequent http 429 error responses?** GoToSocial is configured to use per-IP [rate limiting](./api/ratelimiting.md) by default, but in certain situations it can't accurately identify the remote IP and will treat all connections as coming from the same place. In those cases, the rate limiting needs to be disabled or reconfigured.
+Unlike Mastodon, the default post visibility is Unlisted. If you want something to be visible on your profile page, the post must have Public visibility.
 
-- **Why am I getting frequent http 503 error responses?** Code 503 is returned to callers when your instance is under heavy load and requests are being throttled. This behavior can be tuned as desired, or turned off entirely, see [here](./api/throttling.md).
+## Why aren't my posts showing up on other servers?
 
-- **I keep getting a 400 Bad Request error, and I have done everything suggested by the error message. What should I do?** Verify that the `host` configuration matches the domain that GoToSocial is served from (the domain that users use to acces the server).
+First check the visibility as noted above. TODO: explain how to debug common federation issues
 
-- **My instance is deployed and I'm logged in to a client but my timelines are empty, what's up there?** To see posts, you have to start following people! Once you've followed a few people and they've posted or boosted things, you'll start seeing them in your timelines. Right now GoToSocial doesn't have a way of 'backfilling' posts -- that is, fetching previous posts from other instances -- so you'll only see new posts of people you follow. If you want to interact with an older post of theirs, you can copy the link to the post from their web profile, and paste it in to your client's search bar.
+## Why am I getting frequent http 429 error responses?
 
-- **How can I sign up for a server?** Right now the only way to create an account is by the server's admin to run a command directly on the server. A web-based signup flow is in the roadmap but not implemented yet.
+GoToSocial is configured to use per-IP [rate limiting](./api/ratelimiting.md) by default, but in certain situations it can't accurately identify the remote IP and will treat all connections as coming from the same place. In those cases, the rate limiting needs to be disabled or reconfigured.
 
-- **Why's it still in alpha?** Take a look at the [list of open bugs](https://github.com/superseriousbusiness/gotosocial/issues?q=is%3Aissue+is%3Aopen+label%3Abug) and the [roadmap](https://github.com/superseriousbusiness/gotosocial/blob/main/ROADMAP.md) for a more detailed rundown, but the main missing features at the time of this writing are:
-    * muting conversations
-    * backfill of posts
-    * web-based signup
-    * profile metadata fields
-    * lists of users
-    * polls
-    * scheduling posts
-    * account migration
-    * federated hashtag search
-    * shared block lists across servers
+## Why am I getting frequent HTTP 503 error responses?
+
+Code 503 is returned to callers when your instance is under heavy load and requests are being throttled. This behavior can be tuned as desired, or turned off entirely, see [here](./api/throttling.md).
+
+## I keep getting a 400 Bad Request error, and I have done everything suggested by the error message. What should I do?
+
+Verify that the `host` configuration matches the domain that GoToSocial is served from (the domain that users use to acces the server).
+
+## I keep seeing 'dial within blocked / reserved IP range' in my server logs, and I can't connect to some instances from my instance, what do I do?
+
+The IP address of the remote instance may be in one of the blocked "special use" IP ranges hardcoded into GoToSocial for security reasons. If you need to, you can override this in your configuration file. Have a look at the [http client docs](./configuration/httpclient.md) for this, and please read the warnings there carefully! If you add an explicit allow, you will have to restart your GoToSocial instance to make the config take effect.
+
+## My instance is deployed and I'm logged in to a client but my timelines are empty, what's up there?
+
+To see posts, you have to start following people! Once you've followed a few people and they've posted or boosted things, you'll start seeing them in your timelines. Right now GoToSocial doesn't have a way of 'backfilling' posts -- that is, fetching previous posts from other instances -- so you'll only see new posts of people you follow. If you want to interact with an older post of theirs, you can copy the link to the post from their web profile, and paste it in to your client's search bar.
+
+## How can I sign up for a server?
+
+Right now the only way to create an account is by the server's admin to run a command directly on the server. A web-based signup flow is in the roadmap but not implemented yet.
+
+## Why's it still in alpha?
+
+Take a look at the [list of open bugs](https://github.com/superseriousbusiness/gotosocial/issues?q=is%3Aissue+is%3Aopen+label%3Abug) and the [roadmap](https://github.com/superseriousbusiness/gotosocial/blob/main/ROADMAP.md) for a more detailed rundown, but the main missing features at the time of this writing are:
+
+- muting conversations
+- backfill of posts
+- web-based signup
+- polls
+- scheduling posts
+- account migration
+- federated hashtag search
+- shared block lists across servers

example/config.yaml

@@ -813,6 +813,57 @@ tracing-endpoint: ""
 # Default: false
 tracing-insecure-transport: false
 
+################################
+##### HTTP CLIENT SETTINGS #####
+################################
+
+# Settings for OUTGOING http client connections used by GoToSocial to make
+# requests to remote resources (status GETs, media GETs, inbox POSTs, etc).
+
+http-client:
+
+  # Duration. Timeout to use for outgoing HTTP requests. If the timeout
+  # is exceeded, the connection to the remote server will be dropped.
+  # A value of 0s indicates no timeout: this is not advised!
+  # Examples: ["5s", "10s", "0s"]
+  # Default: "10s"
+  timeout: "10s"
+
+  ########################################
+  #### RESERVED IP RANGE EXCEPTIONS ######
+  ########################################
+  #
+  # Explicitly allow or block outgoing dialing within the provided IPv4/v6 CIDR ranges.
+  #
+  # By default, as a basic security precaution, GoToSocial blocks outgoing dialing within most "special-purpose"
+  # IP ranges. However, it may be desirable for admins with more exotic setups (proxies, funky NAT, etc) to
+  # explicitly override one or more of these otherwise blocked ranges.
+  #
+  # Each of the below allow/block config options accepts an array of IPv4 and/or IPv6 CIDR strings.
+  # For example, to override the hardcoded block of IPv4 and IPv6 dialing to localhost, set:
+  #
+  #   allow-ips: ["127.0.0.1/32", "::1/128"].
+  #
+  # You can also use YAML multi-line arrays to define these, but be diligent with indentation.
+  #
+  # When dialing, GoToSocial will first check if the destination falls within explicitly allowed IP ranges,
+  # then explicitly blocked IP ranges, then the default (hardcoded) blocked IP ranges, returning OK on the
+  # first allowed match, not OK on the first blocked match, or just defaulting to OK if nothing is matched.
+  #
+  # As with all security settings, it is better to start too restrictive and then ease off depending on
+  # your use case, than to start too permissive and try to close the stable door after the horse has
+  # already bolted. With this in mind:
+  # - Don't touch these settings unless you have a good reason to, and only if you know what you're doing.
+  # - When adding explicitly allowed exceptions, use the narrowest possible CIDR for your use case.
+  #
+  # For reserved / special ranges, see:
+  # - https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml
+  # - https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml
+  #
+  # Both allow-ips and block-ips default to an empty array.
+  allow-ips: []
+  block-ips: []
+
 #############################
 ##### ADVANCED SETTINGS #####
 #############################

internal/config/config.go

@@ -150,6 +150,9 @@ type Configuration struct {
 	AdvancedThrottlingRetryAfter time.Duration `name:"advanced-throttling-retry-after" usage:"Retry-After duration response to send for throttled requests."`
 	AdvancedSenderMultiplier     int           `name:"advanced-sender-multiplier" usage:"Multiplier to use per cpu for batching outgoing fedi messages. 0 or less turns batching off (not recommended)."`
 
+	// HTTPClient configuration vars.
+	HTTPClient HTTPClientConfiguration `name:"http-client"`
+
 	// Cache configuration vars.
 	Cache CacheConfiguration `name:"cache"`
 
@@ -163,6 +166,12 @@ type Configuration struct {
 	RequestIDHeader string `name:"request-id-header" usage:"Header to extract the Request ID from. Eg.,'X-Request-Id'."`
 }
 
+type HTTPClientConfiguration struct {
+	AllowIPs []string      `name:"allow-ips"`
+	BlockIPs []string      `name:"block-ips"`
+	Timeout  time.Duration `name:"timeout"`
+}
+
 type CacheConfiguration struct {
 	GTS GTSCacheConfiguration `name:"gts"`
 

internal/config/defaults.go

@@ -208,6 +208,12 @@ var Defaults = Configuration{
 		VisibilitySweepFreq: time.Minute,
 	},
 
+	HTTPClient: HTTPClientConfiguration{
+		AllowIPs: make([]string, 0),
+		BlockIPs: make([]string, 0),
+		Timeout:  10 * time.Second,
+	},
+
 	AdminMediaPruneDryRun: true,
 
 	RequestIDHeader: "X-Request-Id",

internal/config/flags.go

@@ -55,6 +55,11 @@ func (s *ConfigState) AddGlobalFlags(cmd *cobra.Command) {
 		cmd.PersistentFlags().String(DbSqliteSynchronousFlag(), cfg.DbSqliteSynchronous, fieldtag("DbSqliteSynchronous", "usage"))
 		cmd.PersistentFlags().Uint64(DbSqliteCacheSizeFlag(), uint64(cfg.DbSqliteCacheSize), fieldtag("DbSqliteCacheSize", "usage"))
 		cmd.PersistentFlags().Duration(DbSqliteBusyTimeoutFlag(), cfg.DbSqliteBusyTimeout, fieldtag("DbSqliteBusyTimeout", "usage"))
+
+		// HTTPClient
+		cmd.PersistentFlags().StringSlice(HTTPClientAllowIPsFlag(), cfg.HTTPClient.AllowIPs, "no usage string")
+		cmd.PersistentFlags().StringSlice(HTTPClientBlockIPsFlag(), cfg.HTTPClient.BlockIPs, "no usage string")
+		cmd.PersistentFlags().Duration(HTTPClientTimeoutFlag(), cfg.HTTPClient.Timeout, "no usage string")
 	})
 }
 

internal/config/gen/gen.go

@@ -96,16 +96,22 @@ func generateFields(output io.Writer, prefixes []string, t reflect.Type) {
 		flagPath := strings.Join(append(prefixes, field.Tag.Get("name")), "-")
 		flagPath = strings.ToLower(flagPath)
 
+		// Get type without "config." prefix.
+		fieldType := strings.ReplaceAll(
+			field.Type.String(),
+			"config.", "",
+		)
+
 		// ConfigState structure helper methods
 		fmt.Fprintf(output, "// Get%s safely fetches the Configuration value for state's '%s' field\n", name, fieldPath)
-		fmt.Fprintf(output, "func (st *ConfigState) Get%s() (v %s) {\n", name, field.Type.String())
+		fmt.Fprintf(output, "func (st *ConfigState) Get%s() (v %s) {\n", name, fieldType)
 		fmt.Fprintf(output, "\tst.mutex.Lock()\n")
 		fmt.Fprintf(output, "\tv = st.config.%s\n", fieldPath)
 		fmt.Fprintf(output, "\tst.mutex.Unlock()\n")
 		fmt.Fprintf(output, "\treturn\n")
 		fmt.Fprintf(output, "}\n\n")
 		fmt.Fprintf(output, "// Set%s safely sets the Configuration value for state's '%s' field\n", name, fieldPath)
-		fmt.Fprintf(output, "func (st *ConfigState) Set%s(v %s) {\n", name, field.Type.String())
+		fmt.Fprintf(output, "func (st *ConfigState) Set%s(v %s) {\n", name, fieldType)
 		fmt.Fprintf(output, "\tst.mutex.Lock()\n")
 		fmt.Fprintf(output, "\tdefer st.mutex.Unlock()\n")
 		fmt.Fprintf(output, "\tst.config.%s = v\n", fieldPath)
@@ -117,8 +123,8 @@ func generateFields(output io.Writer, prefixes []string, t reflect.Type) {
 		fmt.Fprintf(output, "// %sFlag returns the flag name for the '%s' field\n", name, fieldPath)
 		fmt.Fprintf(output, "func %sFlag() string { return \"%s\" }\n\n", name, flagPath)
 		fmt.Fprintf(output, "// Get%s safely fetches the value for global configuration '%s' field\n", name, fieldPath)
-		fmt.Fprintf(output, "func Get%[1]s() %[2]s { return global.Get%[1]s() }\n\n", name, field.Type.String())
+		fmt.Fprintf(output, "func Get%[1]s() %[2]s { return global.Get%[1]s() }\n\n", name, fieldType)
 		fmt.Fprintf(output, "// Set%s safely sets the value for global configuration '%s' field\n", name, fieldPath)
-		fmt.Fprintf(output, "func Set%[1]s(v %[2]s) { global.Set%[1]s(v) }\n\n", name, field.Type.String())
+		fmt.Fprintf(output, "func Set%[1]s(v %[2]s) { global.Set%[1]s(v) }\n\n", name, fieldType)
 	}
 }