Commit Graph

712 Commits

Author SHA1 Message Date
ThibG 6b0eda14a1 Avoid race condition when streaming deleted statuses (#10280)
* Avoid race condition when streaming deleted statuses

* Move redis lock to DistributionWorker to avoid extra Redis value
2019-03-16 20:18:47 +01:00
Eugen Rochko 3ad3223b46
Fix detailed poll validation errors not being returned in the API (#10261)
No more "Owned poll is invalid"
2019-03-15 13:36:38 +01:00
Eugen Rochko ba84b6d4d7
Add `visibility` param to reblog REST API (#9851)
Use async worker for creating reblog notification to improve performance
2019-03-15 04:36:41 +01:00
Eugen Rochko 51e154f5e8
Admission-based registrations mode (#10250)
Fix #6856
Fix #6951
2019-03-14 05:28:30 +01:00
Eugen Rochko 9f5b55ad4f
Fix poll update handler calling method was that was not available (#10246)
* Fix poll update handler calling method was that was not available

Fix regression from #10209

* Refactor VoteService

* Refactor ActivityPub::DistributePollUpdateWorker and optimize it

* Fix typo

* Fix typo
2019-03-12 22:58:59 +01:00
Eugen Rochko a41afef80c
Optimize RemoveStatusService by removing n+1 query and using shared inboxes (#10247) 2019-03-12 22:34:55 +01:00
ThibG 3a92885a86 Support pushing and receiving updates to poll tallies (#10209)
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
ThibG c11dff5049 Reject existing Follows when suspending a remote account (#10230)
* Reject existing Follows when suspending a remote account

Partial fix to #10229

* Add tests
2019-03-10 16:18:58 +01:00
ThibG 3aaac4f134 Do not allow adding votes to expired polls (#10214)
* Do not allow adding votes to expired polls

* Only validate expires_at on create
2019-03-08 00:54:50 +01:00
ThibG 09c042aa10 Handle StaleObjectError when retrieving polls (#10208) 2019-03-07 15:52:38 +01:00
ThibG df5924a1db Do not error out on unsalvageable errors in FetchRepliesService (#10175)
* Do not error out on unsalvageable errors in FetchRepliesService

Fixes #10152

* Fix FetchRepliesWorker erroring out on deleted statuses
2019-03-05 15:21:14 +01:00
ThibG d785497ba5 Fix suspended account's fields being set as empty dict instead of list (#10178)
Fixes #10177
2019-03-05 15:19:54 +01:00
Eugen Rochko 4037b5eb1e
Fix last_fetched_at not being set on polls (#10170) 2019-03-05 04:10:01 +01:00
Eugen Rochko e6900b167b
Fix another typo in ActivityPub::FetchRemotePollService (#10146) 2019-03-04 00:52:18 +01:00
Eugen Rochko 878a75ba21
Fix typo in ActivityPub::FetchRemotePollService (#10145) 2019-03-04 00:50:56 +01:00
Eugen Rochko ae1b9cf70a
Fix remote poll expiration time (#10144) 2019-03-04 00:44:34 +01:00
ThibG f821eca3b3 Correctly make polls and media mutually exclusive (#10141) 2019-03-04 00:40:21 +01:00
ThibG e13d3792f3 Make sure the poll is created before storing its id (#10142)
* Make sure the poll is created before storing its id

* Fix updating poll results

* Support fetching Question activities from the search bar
2019-03-04 00:39:06 +01:00
Eugen Rochko 230a012f00
Add polls (#10111)
* Add polls

Fix #1629

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
ThibG 9d3c6f1849 Improved remote thread fetching (#10106)
* Fetch up to 5 replies when discovering a new remote status

This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.

* Add some tests for ActivityPub::FetchRepliesWorker

* Add specs for ActivityPub::FetchRepliesService

* Serialize up to 5 public self-replies for ActivityPub notes

* Add specs for ActivityPub::NoteSerializer

* Move exponential backoff logic to a worker concern

* Fetch first page of paginated collections when fetching thread replies

* Add specs for paginated collections in replies

* Move Note replies serialization to a first CollectionPage

The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.

* Use pluck(:uri) instead of map(&:uri) to improve performances

* Fix fetching replies when they are in a CollectionPage
2019-02-28 15:22:21 +01:00
Eugen Rochko e7f20cc43f
Add type, limit, offset, min_id, max_id, account_id to search API (#10091)
* Add type, limit, offset, min_id, max_id, account_id to search API

Fix #8939

* Make the offset work on accounts and hashtags search as well

* Assure brakeman we are not doing mass assignment here

* Do not allow paginating unless a type is chosen

* Fix search query and index id field on statuses instead of created_at
2019-02-26 15:21:36 +01:00
NOGISAKA Sadata e050958794 Skip some methods in BatchedRemoveStatusService when account is nil (#10095)
* Skip some methods in BatchedRemoveStatusService when account is nil

Sometimes `bin/tootctl accounts cull` fails with below error:

undefined method `followers_for_local_distribution' for nil:NilClass (NoMethodError)

This commit makes BatchedRemoveStatusService to skip below methods when
`account` is nil:

- unpush_from_home_timelines()
- unpush_from_list_timelines()
- batch_stream_entries()

* Fix rubocop error: Use `next` to skip iteration.
2019-02-24 14:28:36 +01:00
Eugen Rochko dad339da6d
Filter incoming Create activities by relation to local activity (#10005)
Reject those from accounts with no local followers, from relays
that are not enabled, which do not address local accounts and are
not replies to accounts that do have local followers
2019-02-13 18:42:47 +01:00
ThibG 99c4b5b75a Move sending account Delete to anyone but the account's followers to the pull̀ queue (#10016) 2019-02-12 05:10:43 +01:00
abcang e186bd2fb0 Fix Tombstone.delete_all ArgumentError (#9978) 2019-02-06 02:50:52 +01:00
Eugen Rochko 364f2ff9aa
Add featured hashtags to profiles (#9755)
* Add hashtag filter to profiles

GET /@:username/tagged/:hashtag
GET /api/v1/accounts/:id/statuses?tagged=:hashtag

* Display featured hashtags on public profile

* Use separate model for featured tags

* Update featured hashtag counters on-write

* Limit featured tags to 10
2019-02-04 04:25:59 +01:00
Eugen Rochko d14c276e58
Add option to overwrite imported data (#9962)
* Add option to overwrite imported data

Fix #7465

* Add import for domain blocks
2019-02-03 03:59:51 +01:00
ysksn bcfff65195 Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
ThibG e2a5be6e9a Prevent posting toots with media attachments from someone else (#9921) 2019-01-26 23:59:39 +01:00
Renato "Lond" Cerqueira e1ec3a9f09 Rescue SSL error in verify link service (#9914)
If the first link to be verified contains a rel=me link with a SSL
error, the VerifyAccountLinksWorker will fail and not try the following
links. This rescues the SSL error when fetching the link, avoiding this
issue.
2019-01-24 15:38:18 +01:00
ThibG 061feb63ed Fix scheduled toot with media immediately creating a toot (#9894)
* Add test for not persisting status when attaching media to scheduled toot

* Prevent status used for validation from being persisted to the database

Fixes #9893

Thanks to tateisu for the help investigating this.
2019-01-21 20:03:04 +01:00
Eugen Rochko 6ca9015789
Fix error when `scheduled_at` attribute is malformed (#9866) 2019-01-19 22:01:01 +01:00
ThibG aeb124491d Reject existing Follow in addition to sending a Block (#9811)
Mastodon expects remote servers to remove follow relationships upon receiving
a Block. However, the spec only evokes Block activities in a C2S context, never
in a S2S context.

This PR, in addition to federating the Block, explicitly sends a Reject for any
affected follow relationship, which makes a bit more sense with regards to the
spec.
2019-01-18 15:57:19 +01:00
ThibG 75b1488cf4 Add tombstones for remote statuses (#9830)
* Add Tombstone model to remember object deletion

* Do not recreate a status if it has been deleted

* Record Tombstone for remote deleted items

Also, only record deleted items from same-host actors

* Clear an user's tombstones when their key change
2019-01-18 15:56:55 +01:00
Eugen Rochko 69f782b54d
Fix code style of regeneration-related code (#9843) 2019-01-18 01:02:51 +01:00
Renato "Lond" Cerqueira 5c5e14c816 Fix undefined method error in sidekiq (#9807)
* Fix undefined method error in sidekiq

Body can be not nil but still be empty, which causes a
`NoMethodError: undefined method `[]' for nil:NilClass` further in the
code. This checks for an empty body to avoid the issue.

* Fix codeclimate issue
2019-01-14 17:28:41 +01:00
ThibG 28b482874a Improvements to signature verification (#9667)
* Refactor signature verification a bit

* Rescue signature verification if recorded public key is invalid

Fixes #8822

* Always re-fetch AP signing key when HTTP Signature verification fails

But when the account is not marked as stale, avoid fetching collections and
media, and avoid webfinger round-trip.

* Apply stoplight to key/account update as well as initial key retrieval
2019-01-07 21:45:13 +01:00
Eugen Rochko 43c61bca60
Add locale param to sign-up API (#9747)
Fix #9627
2019-01-07 14:50:20 +01:00
Eugen Rochko 1cbdf8d218
Fix wrong param name in scheduled statuses and return params in API (#9725)
The database column and API param are called in_reply_to_id, not
in_reply_to_status_id, so it makes no sense to encode it that way
2019-01-06 12:03:27 +01:00
Eugen Rochko a49d43d112
Add scheduled statuses (#9706)
Fix #340
2019-01-05 12:43:28 +01:00
Naoki Kosaka fae32634b1 Use Contact User as Relay, Report, Subscribe. (#9661)
* Use Contact User as Relay, Report, Subscribe.

* Use Account.representative to fetch contact user.

* Use find_local.

* No reason to use Account.representative in subscribe_service.

* Don't required representative!

* Fallback is included in Account.representative method.
2019-01-05 07:17:12 +01:00
kodai 6a090db4c9 Fix links not being verified (#9673)
Fix #9365
2018-12-31 15:29:10 +01:00
Jeong Arm 511231ce57 Skip full text search if failed (#9654)
* Skip full text search if failed

* Move rescue block to "perform_statuses_search!"
2018-12-30 19:00:29 +01:00
ThibG 6fb6a53938 Avoid duplicate work by merging ReplyDistributionWorker into DistributionWorker (#9660) 2018-12-30 19:00:04 +01:00
ThibG 290932602b Reduce usage of LD signatures (#9659)
* Do not LDS-sign Follow, Accept, Reject, Undo, Block

* Do not use LDS for Create activities of private toots

* Minor cleanup

* Ignore unsigned activities instead of misattributing them

* Use status.distributable? instead of querying visibility directly
2018-12-30 09:48:59 +01:00
Eugen Rochko 0f938ff29c
Add handler for Move activity (#9629) 2018-12-29 02:24:36 +01:00
Eugen Rochko 6e49907ecf
Improve admin UI for account view (#9643) 2018-12-28 03:38:41 +01:00
Eugen Rochko 5d2fc6de32
Add REST API for creating an account (#9572)
* Add REST API for creating an account

The method is available to apps with a token obtained via the client
credentials grant. It creates a user and account records, as well as
an access token for the app that initiated the request. The user is
unconfirmed, and an e-mail is sent as usual.

The method returns the access token, which the app should save for
later. The REST API is not available to users with unconfirmed
accounts, so the app must be smart to wait for the user to click a
link in their e-mail inbox.

The method is rate-limited by IP to 5 requests per 30 minutes.

* Redirect users back to app from confirmation if they were created with an app

* Add tests

* Return 403 on the method if registrations are not open

* Require agreement param to be true in the API when creating an account
2018-12-24 19:12:38 +01:00
ThibG 5f387995d9 Limit maximum visibility of local silenced users to unlisted (#9583)
Fixes #9580
2018-12-24 19:06:14 +01:00
ThibG e709b8da0d Ignore low-confidence CharlockHolmes guesses when parsing link cards (#9510)
* Add failing test for windows-1251 link cards

* Ignore low-confidence CharlockHolmes guesses

Fixes #9466

* Fix no method error when charlock holmes cannot detect charset
2018-12-17 19:19:45 +01:00
Sumit Khanna 769c2d2680 Error message for avatar image that's too large. #9204 (#9518)
* Error message for avatar image that's too large. #9204

* Code climate/formatting

* Removed avatar error message

* Moved valid image dimentions check to update service

* removed unnescessary begin block

* code climate formatting

* code climate indent fix
2018-12-14 05:07:21 +01:00
ysksn 83975a2394 Remove redundant (#9503) 2018-12-12 05:54:34 +01:00
ysksn 351ed12b2f Use find_or_initialize_by (#9504) 2018-12-12 05:54:09 +01:00
Eugen Rochko 73be8f38c1
Add profile directory (#9427)
Fix #5578
2018-12-06 17:36:11 +01:00
Eugen Rochko 6ddf0432e7
Improve account suspension speed and completeness (#9290)
- Some associations were missing from the clean-up
- Some attributes were not reset on suspension
- Skip federation and streaming deletes when purging a dead domain
- Move account association definitions to concern
2018-12-03 01:32:08 +01:00
valerauko 824497fbce Ignore JSON-LD profile in mime type comparison (#9179)
Ignore JSON-LD profile in mime type comparison
2018-11-22 12:49:07 +01:00
James Kiesel 6a65d87f4d Simplify hashtag query service, perf increase for 'any' tags (#9298) 2018-11-19 12:49:50 +01:00
Eugen Rochko 8069fd636b
Remove intermediary arrays when creating hash maps from results (#9291) 2018-11-16 15:02:18 +01:00
ThibG 7f5a4be580 Fix emoji update date processing (#9255) 2018-11-10 23:59:51 +01:00
ThibG 9b89c62d43 Check that twitter:player is valid before using it (#9254)
Fixes #9251
2018-11-10 20:42:04 +01:00
Eugen Rochko 6d59dfa15d
Optimize the process of following someone (#9220)
* Eliminate extra accounts select query from FollowService

* Optimistically update follow state in web UI and hide loading bar

Fix #6205

* Asynchronize NotifyService in FollowService

And fix failing test

* Skip Webfinger resolve routine when called from FollowService if possible

If an account is ActivityPub, then webfinger re-resolving is not necessary
when called from FollowService. Improve options of ResolveAccountService
2018-11-08 21:05:42 +01:00
m.b cd41c2c6ad Update resolve_url_service.rb (#9188) 2018-11-05 18:54:07 +01:00
James Kiesel 4c03e05a4e Allow joining several hashtags in a single column (#8904)
* Nascent tag menu on frontend

* Hook up frontend to search

* Tag intersection backend first pass

* Update yarnlock

* WIP

* Fix for tags not searching correctly

* Make radio buttons function

* Simplify radio buttons with modeOption

* Better naming

* Rearrange options

* Add all/any/none functionality on backend

* Small PR cleanup

* Move to service from scope

* Small cleanup, add proper service tests

* Don't use send with user input :D

* Set appropriate column header

* Handle auto updating timeline

* Fix up toggle function

* Use tag value correctly

* A bit more correct to use 'self' rather than 'all' in status scope

* Fix some style issues

* Fix more code style issues

* Style select dropdown more better

* Only use to_id'ed value to ensure no SQL injection

* Revamp frontend to allow for multiple selects

* Update backend / col header to account for more flexible tagging

* Update brakeman ignore

* Codeclimate suggestions

* Fix presenter tag_url

* Implement initial PR feedback

* Handle additional tag streaming

* CodeClimate tweak
2018-11-05 18:53:25 +01:00
Eugen Rochko ce2ee68b64
Revert "Fix FetchAtomService content type handling (#9132)" (#9171)
This reverts commit c36a4a1617.
2018-10-31 00:43:34 +01:00
valerauko c36a4a1617 Fix FetchAtomService content type handling (#9132)
* Add profile to json+ld in Accept

It's required by the ActivityPub spec

* Use headers['Content-type'] instead of mime_type

mime_type strips the profile from the content type, but it's still available raw in the headers hash

* Add test for ld+json with profile
2018-10-30 15:07:57 +01:00
Eugen Rochko be202f9377
Accept the same payload in multiple inboxes and deliver (#9150) 2018-10-30 15:03:55 +01:00
Eugen Rochko 47b8d195e6
Always let through notifications from staff (#9152)
* Always let through notifications from staff

Follow-up to #8993

* Let messages from staff through, but no other notifications
2018-10-30 15:02:55 +01:00
ThibG a03d506626 Fix Pleroma mentions being fetched as preview cards (#9158) 2018-10-30 15:02:24 +01:00
ThibG 33a71e8f7c Do not hide boost notifications from followed people with hidden boosts (#9147)
* Do not hide boost notifications from followed people with hidden boosts

Not displaying boosts from a followed user in the Home timeline and not
having notifications when they reblog your own content are two very
separate concerns, tying them together seem counter-intuitive and unwanted.

* Update specs accordingly
2018-10-30 00:47:31 +01:00
Eugen Rochko 795f0107d2
Include preview cards in status entity in REST API (#9120)
* Include preview cards in status entity in REST API

* Display preview card in-stream

* Improve in-stream display of preview cards
2018-10-28 06:35:03 +01:00
Eugen Rochko eef8d9a5f7
Add locality check to ActivityPub::FetchRemoteAccountService (#9109)
* Add locality check to ActivityPub::FetchRemoteAccountService

Fix #8643

Because there are a few places where it is called, it is difficult
to confirm if they all previously checked it for locality. It's better
to make sure within the service.

* Remove faux-remote duplicates of local accounts
2018-10-26 23:08:34 +02:00
abcang 1e2695198a Skip link-back check if body is nil (#9107) 2018-10-26 03:31:23 +02:00
ThibG 7fee968e9f Do not fetch preview card for mentioned users (#6934) 2018-10-25 18:13:19 +02:00
Eugen Rochko ddd30f331c
Improve support for aspects/circles (#8950)
* Add silent column to mentions

* Save silent mentions in ActivityPub Create handler and optimize it

Move networking calls out of the database transaction

* Add "limited" visibility level masked as "private" in the API

Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic

* Ensure silent column is non-null, add spec

* Ensure filters don't check silent mentions for blocks/mutes

As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering

* Clean up code

* Use Status#active_mentions to limit returned mentions

* Fix code style issues

* Use Status#active_mentions in Notification

And remove stream_entry eager-loading from Notification
2018-10-17 17:13:04 +02:00
Quint Guvernator f5e2e96e95 always allow DMs from staff (#8993) 2018-10-16 19:55:05 +02:00
Eugen Rochko 87fdd139b8
Do not push DMs into the home feed (#8940)
* Do not push DMs into the home feed

* Show DMs column after sending a DM, if DMs column is not already shown
2018-10-11 01:31:03 +02:00
Eugen Rochko 774ac47373
Add conversations API (#8832)
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Eugen Rochko 7fe137d2f7
Fix link verification for remote accounts (#8868) 2018-10-04 15:47:03 +02:00
Eugen Rochko f92f1ee80a
Support link verification with redirects (#8735)
(e.g. URL shortener)
2018-09-20 00:10:35 +02:00
Yamagishi Kazutoshi 3da1cc7d5e Fix failed profile verification when rel attribute including values other than me (#8733) 2018-09-19 16:47:31 +02:00
Eugen Rochko 68833a50d4
Fix VerifyAccountLinksWorker not being queued (#8721)
UX-wise, people expect that saving the profile will re-check links even without changing fields content. Bug-wise, `@account` was undefined.

Regression from #8703
2018-09-18 23:57:21 +02:00
Eugen Rochko f4d549d300
Redesign forms, verify link ownership with rel="me" (#8703)
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
2018-09-18 16:45:58 +02:00
ThibG bd9e47e9be Handle relative URLs when fetching OEmbed/OpenGraph cards (#8669) 2018-09-10 18:26:28 +02:00
Eugen Rochko c593d6df9c
Add preference for report notification e-mails, skip for duplicates (#8559)
If an unresolved report for the same target account already exists,
no new notification is generated
2018-09-02 00:11:58 +02:00
Eugen Rochko cabdbb7f9c
Add CLI task for rotating keys (#8466)
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
Eugen Rochko 8adf485c0f
Bump interactions count when a follow happens (#8468) 2018-08-26 20:02:45 +02:00
Eugen Rochko 55c856c522
Delay e-mail notifications by 2 minutes (#8438)
Fix #8430
2018-08-26 16:53:19 +02:00
ThibG 42573b76f1 Do not crash if remote custom emoji does not define updated date (fixes #8376) (#8377) 2018-08-23 00:27:58 +02:00
Eugen Rochko 802cf6a4c5
Improve federated ID validation (#8372)
* Fix URI not being sufficiently validated with prefetched JSON

* Add additional id validation to OStatus documents, when possible
2018-08-22 20:55:14 +02:00
Eugen Rochko 1b282299df
Add favourites to archive takeout (#8351)
Remove experimental key export
2018-08-22 19:33:10 +02:00
Eugen Rochko d98de8ada7
Get rid of all batch order warnings (#8334) 2018-08-21 12:25:50 +02:00
Eugen Rochko 6226aa83d7
Increase reach of Delete->Actor activities (#8305)
Fix #7316
2018-08-20 13:28:05 +02:00
ThibG abc5548cca Do not process outgoing mentions to suspended accounts (#8272) 2018-08-18 19:42:13 +02:00
ThibG 1ee675d68b Use correct activity id in Accept when receiving duplicate Follow (fixes #8218) (#8244) 2018-08-17 14:08:17 +02:00
ThibG af912fb308 Allow accessing local private/DM messages by URL (#8196)
* Allow accessing local private/DM messages by URL

(Provided the user pasting the URL is authorized to see the toot, obviously)

* Fix SearchServiceSpec tests
2018-08-15 19:33:36 +02:00
Eugen Rochko cc94b1d95a Prevent ActivityPub movedTo recursion (#8092)
Fix #8051
2018-07-28 17:14:55 -04:00
Eugen Rochko a23ac107e4
Fix activity:interactions counter to count all interactions (#8037) 2018-07-16 18:35:43 +02:00
Eugen Rochko e55dce3176
Add federation relay support (#7998)
* Add federation relay support

* Add admin UI for managing relays

* Include actor on relay-related activities

* Fix i18n
2018-07-13 02:16:06 +02:00
Eugen Rochko 8461cd4bb5
Send undo of boost to original poster if reblog (#7959)
Fix #7874
2018-07-05 20:57:50 +02:00
Eugen Rochko b0968623fa
Fix check for PotentialFriendshipTracker when replying (#7933) 2018-07-03 05:00:44 +02:00