Add a config option to block "expensive" pages for anonymous users (#34024) (#34071)

Backport #34024 since there are too many AI crawlers. The new code is
covered by tests and it does nothing if users don't set it.

modules/setting/config_provider.go

@@ -26,6 +26,7 @@ type ConfigKey interface {
 	In(defaultVal string, candidates []string) string
 	String() string
 	Strings(delim string) []string
+	Bool() (bool, error)
 
 	MustString(defaultVal string) string
 	MustBool(defaultVal ...bool) bool