Commit Graph

10790 Commits

Author SHA1 Message Date
rinpatch 2ce141fb0e Merge branch 'fix/oban-pruning' into 'develop'
Bring back oban job pruning

Closes #1945

See merge request pleroma/pleroma!2768
2020-07-15 13:52:01 +00:00
feld a2f9bc77b6 Merge branch 'issue/sign-authority' into 'develop'
Include port in host for signatures

See merge request pleroma/pleroma!2766
2020-07-15 13:44:52 +00:00
Haelwenn (lanodan) Monnier d29b8997f4
MastoAPI: fix & test giving MRF reject reasons 2020-07-15 15:25:33 +02:00
rinpatch c413649a8d Bring back oban job pruning
Closes #1945
2020-07-15 16:20:17 +03:00
rinpatch 7115c5f82e ConnectionPool.Worker: do not stop with an error when there is a timeout
This produced error log messages about GenServer termination
every time the connection was not open due to a timeout.

Instead we stop with `{:shutdown, <gun_error>}` since shutting down
when the connection can't be established is normal behavior.
2020-07-15 15:58:08 +03:00
Maksim Pechnikov 3f8bff8d6b Merge branch 'develop' into issue/sign-authority 2020-07-15 15:49:46 +03:00
href 6d583bcc3b Set a default timeout for Gun adapter timeout 2020-07-15 15:26:35 +03:00
href e499275076 Don't test tls_options in adapter helper test. 2020-07-15 15:26:35 +03:00
href afd378f84c host is now useless 2020-07-15 15:26:35 +03:00
href ce1a42bd04 Simplify TLS opts
- `verify_fun` is not useful now
- use `customize_check_hostname` (OTP 20+ so OK)
- `partial_chain` is useless as of OTP 21.1 (wasn't there, but hackney/..
uses it)
2020-07-15 15:26:35 +03:00
href ebfa591689 Go back to upstream Tesla 2020-07-15 15:26:35 +03:00
href 6b1f6a1cf7 Bump gun 2020-07-15 15:26:35 +03:00
href 53ba6815b1 parentheses... 2020-07-15 15:26:35 +03:00
href 23d714ed30 Fix race in enforcer/reclaimer start 2020-07-15 15:26:35 +03:00
href 6a0f2bdf8c Ensure connections error get known by the caller 2020-07-15 15:26:35 +03:00
href 46dd276d68 ConnectionPool.Worker: Open gun conn in continue instead of init 2020-07-15 15:26:35 +03:00
rinpatch 33747e9366 config.exs: set gun retries to 0
The new pooling code just removes the connection when it's down,
there is no need to reconnect a connection that is just sitting idle,
better just open a new one next time it's needed
2020-07-15 15:26:35 +03:00
rinpatch a705637dcf Connection Pool: fix LRFU implementation to not actually be LRU
The numbers of the native time unit were so small the CRF was always 1,
making it an LRU. This commit switches the time to miliseconds and changes
the time delta multiplier to the one yielding mostly highest hit rates according
to the paper
2020-07-15 15:26:35 +03:00
rinpatch 9b73c35ca8 Request limiter setup: consider {:error, :existing} a success
When the application restarts (which happens after certain config
changes), the limiters are not destroyed, so `ConcurrentLimiter.new`
will produce {:error, :existing}
2020-07-15 15:26:35 +03:00
rinpatch 12fa5541f0 FollowRedirects: Unconditionally release the connection if there is an error
There is no need for streaming the body if there is no body
2020-07-15 15:26:35 +03:00
rinpatch 9df5918974 config.exs: make gun the default again 2020-07-15 15:26:35 +03:00
rinpatch 37f1e781cb Gun adapter helper: fix wildcard cert issues on OTP 23
See https://bugs.erlang.org/browse/ERL-1260 for more info.

The ssl match function is basically copied from mint, except
that `:string.lowercase/1` was replaced by `:string.casefold`.
It was a TODO in mint's code, so might as well do it since we don't need
to support OTP <20.

Closes #1834
2020-07-15 15:26:35 +03:00
rinpatch 007843b75e Add documentation for new connection pool settings and remove some
`:retry_timeout` and `:retry` got removed because reconnecting on failure is
something the new pool intentionally doesn't do.

`:max_overflow` had to go in favor of `:max_waiting`, I didn't reuse the key because
the settings are very different in their behaviour.

`:checkin_timeout` got removed in favor of `:connection_acquisition_wait`,
I didn't reuse the key because the settings are somewhat different.

I didn't do any migrations/deprecation warnings/changelog entries because
these settings were never in stable.
2020-07-15 15:26:35 +03:00
rinpatch 7882f28569 Use erlang monotonic time for CRF calculation 2020-07-15 15:26:35 +03:00
rinpatch 00926a63fb Adapter Helper: Use built-in ip address type 2020-07-15 15:26:35 +03:00
rinpatch bf3492ceb3 Connection Pool: add tests 2020-07-15 15:26:35 +03:00
rinpatch 721e89e88b Remove tests for old pool 2020-07-15 15:26:35 +03:00
rinpatch 4128e3a84a HTTP: Implement max request limits 2020-07-15 15:26:35 +03:00
rinpatch 94c8f3cfaf Use a custom pool-aware FollowRedirects middleware 2020-07-15 15:26:35 +03:00
rinpatch 281ddd5e37 Connection pool: fix connections being supervised by gun_sup 2020-07-15 15:26:35 +03:00
rinpatch 1b15cb066c Connection pool: Add client death tracking
While running this in production I noticed a number of ghost
processes with all their clients dead before they released the connection,
so let's track them to log it and remove them from clients
2020-07-15 15:26:35 +03:00
rinpatch e94ba05e52 Connection pool: Fix a possible infinite recursion if the pool is exhausted 2020-07-15 15:26:35 +03:00
rinpatch 7738fbbaf5 Connection pool: implement logging and telemetry events 2020-07-15 15:26:25 +03:00
rinpatch 0ffde499b8 Connection Pool: register workers using :via 2020-07-15 15:24:47 +03:00
rinpatch ec9d0d146b Connection pool: Fix race conditions in limit enforcement
Fixes race conditions in limit enforcement by putting worker processes
in a DynamicSupervisor
2020-07-15 15:17:27 +03:00
rinpatch d08b157699 Connection pool: check that there actually is a result
Sometimes connections died before being released to the pool, resulting
in MatchErrors
2020-07-15 15:17:27 +03:00
rinpatch fffbcffb8c Connection Pool: don't enforce pool limits if no new connection needs to be opened 2020-07-15 15:17:27 +03:00
rinpatch 58a4f350a8 Refactor gun pooling and simplify adapter option insertion
This patch refactors gun pooling to use Elixir process registry and
simplifies adapter option insertion.

Having the pool use process registry instead of a GenServer has a number of advantages:
- Simpler code: the initial implementation adds about half the lines of code it deletes
- Concurrency: unlike a GenServer, ETS-based registry can handle multiple checkout/checkin
requests at the same time
- Precise and easy idle connection clousure: current proposal for closing idle connections in
the GenServer-based pool needs to filter through all connections once a minute and compare their
last active time with closing time. With Elixir process registry this can be done
by just using `Process.send_after`/`Process.cancel_timer` in the worker process.
- Lower memory footprint: In my tests `gun-memory-leak` branch uses about 290mb on peak load (250 connections)
and 235mb on idle (5-10 connections). Registry-based pool uses 210mb on idle and 240mb on peak load
2020-07-15 15:17:27 +03:00
lain 040524c09f Merge branch 'tests/apc2s-update_outbox' into 'develop'
AP C2S tests: Make sure you can't use another user's AP id

See merge request pleroma/pleroma!2767
2020-07-15 10:12:44 +00:00
Haelwenn (lanodan) Monnier 6d8427cca2
AP C2S tests: Make sure you can't use another user's AP id 2020-07-15 09:59:24 +02:00
Maksim Pechnikov 1dd767b8c7 Include port in host for signatures 2020-07-14 21:44:08 +03:00
Mark Felder 777a7edc6b Lint and fix test to match new log message 2020-07-14 13:15:37 -05:00
Mark Felder 37297a8482 Improve error messages 2020-07-14 13:12:16 -05:00
feld 3f65f2ea79 Merge branch 'feature/1922-media-proxy-whitelist' into 'develop'
Support for hosts with scheme in MediaProxy whitelist setting

Closes #1922

See merge request pleroma/pleroma!2754
2020-07-14 18:07:44 +00:00
feld 63798e89de Merge branch 'fix/scrubber-recompile-warnings-as-errors' into 'develop'
Fix in-db configuration in dev environment

See merge request pleroma/pleroma!2764
2020-07-14 18:05:36 +00:00
feld 2909dc873b Merge branch '1940-admin-token-oauthless-auth' into 'develop'
[#1940] Reinstated OAuth-less `admin_token` authentication

Closes #1940

See merge request pleroma/pleroma!2760
2020-07-14 16:48:26 +00:00
Ivan Tashkinov 124b4709dc [#1940] Added `admin_token` param (as `admin_api_params/0`) to existing Admin API OpenAPI operations. 2020-07-14 19:31:05 +03:00
Mark Felder ce314e6fe2 Clarify description and suggestion 2020-07-14 11:24:58 -05:00
feld 246f49d685 Merge branch '1916-transmogrifier-skipping-oddities-in-address-fields' into 'develop'
Transmogrifier: filtering weirdness in address fields

Closes #1916

See merge request pleroma/pleroma!2762
2020-07-14 15:17:25 +00:00
rinpatch 2d5e96ea8b Merge branch 'fix/oban-2.0-in-db-config' into 'develop'
Migrate in-db config after updating to Oban 2.0

See merge request pleroma/pleroma!2765
2020-07-14 10:39:08 +00:00