Ignore gui - #7145

Looks nice!

So I haven't looked at the code, but I always felt that the only sensible way to do this would be a tree view, as otherwise there will be tons of clicking on the breadcrumb line just to navigate. What's your feeling?

This table+breadcrumbs UI just developed organically as I hacked at this. Tree requires a more complex structure of files, or loading them all at once, and seemed limiting when this might require some unusual button behavior. At least those were the assumptions I made without having tried to use fancytree :) I could be convinced it's the superior method!

Here's the current state of the change, viewing and modifying simple ignore patterns. I've updated some out of date github notification settings so will be responding more promptly.

@calmh @imsodin wdyt?

Fancy. Looks reasonable to me. I'd probably put the patterns above the "tree" (or whatever we call it) widget, as the pattern text box is fixed size and the tree widget can have unbounded height? (Didn't look at any code either.)

You a wizard? :)

The current breadcrumbs approach seems quite usable to me. I'd probably prefer a tree too - less up-and-down mouse movement (even though I wouldn't use the mouse :P ). And a very cursory glance at fancytree suggests it does what we need: tri-state selection and lazy-loading. However it might have other drawbacks and increased complexity. Basically with my current, limited knowledge of this I consider both approaches viable.

Some points I noted on a very cursory glance at the code:

• I didn't see any ordering logic. E.g. if I first include "/foo/bar" and then ignore /foo, it looks like /foo will be inserted first. Which makes the latter !/foo/bar ineffective.

• Due to problems like the above, I believe it's really hard to let the same patterns be used for manual patterns and to be controlled through this new GUI. There have been proposals of "comment fences", to separate the automagically managed patterns from manual once. Maybe that's helpful/needed here - I'll need to have a close look at the logic (and maybe add some nasty test-cases - great to see tests :) ) to get a clearer picture for myself.

Not a wizard just an unemployed web dev

OK I'll continue with the breadcrumbs UI for now, but will experiment with fancytree and see if the same features are compatible. To @imsodin 's points:

1. Yes, it is naive in this respect can make some non-obvious changes. It should be possible to insert them with respect to specificity - I'll add a test :)
2. I plan to add a switch between basic (browser) and advanced (textarea) modes as discussed in lib/ignore Have a nice treeview for handling ignores - Next Gen Ignores #5132, showing only one at a time and disabling access to the basic view entirely when advanced patterns are present. I think that addresses your concern with manual patterns? It hides the text box for scroll concerns, ensures we only modify pattern set when they're simple enough to understand (no need for comment fence), and avoids problems with rendering them in the browse.

Here is what that advanced mode toggle looks like.

I will try out fancytree later this week. Otherwise this is feature complete as much as I planned it and my remaining todo items are small - css, error messaging, translations.

I do have some general questions:

1. What's up with untrusted/? Lots of similar files. Do I need to make changes there?
2. Will I need to take any action to get strings into the catalog/transifex?
3. Just noticed the pathSeparator value in the API. Are there problems on some platforms with my using '/' for paths?

I have not tested this yet, but how do you deal with a lot of folders/files in a folder? Are all of them displayed at once?

What happens if you have e.g. 10,000 files in a folder? Is the GUI able to withstand displaying all that at once? If yes, then what is the RAM overhead?

Here is the change using fancytree.
+ Maintains tree structure without having to keep a matching data structure ourselves. yay!
+ Should have better performance out of the box when making changes to ignores
- Model is not bound to angular/view state, so it's tricky to integrate with other parts of the app. Might be prone to bugs but that concern may be unfounded. (for example, I'm having difficulty resetting the tree when changing folder edit modals)
- Less unit testable - but practically only the toggle and rendering functions are uncovered
So this seems just fine if you'd prefer the tree UI. I still need to fix the bug mentioned above.

Looks great. I know there were some issues with resetting the trees state that I had elsewhere, I think in version restore UI, so perhaps there is something there to be inspired by.

Also, I think fancy tree was just a suggestion of something we used before that seem to have worked, but I think you have more expertise in the UI area than all of the maintainers put together, so perhaps there are better, angular specific components that we could use.

I guess we use an ancient version of angular so it might be hard to find something that works, but I know for sure that angular material has something like this that would work.

One thing that looks strange in the example is the fact that the checkboxes are left aligned, rather than aligned with the item, but I guess that's how fancy tree renders it?

I guess if you feel it's polished enough, let us know, and we'll start digging through the code with our limited UI experience.

Updated for new db/browse response structure.

Re:future Yes comment fence could be used one day. The complexity of handling patterns like globs or extensions comes from the UI making unintended changes, having widespread in the syncthing folder beyond the specific file the user is toggling. I can imagine one day adding a dialog for these patterns to confirm changes, explaining the impact. Or to confirm your change to parent will remove X number of child patterns. Or option to ignore other files with the same name or extension as the current one. Just speculating.

@imsodin should we just merge this?

I have just done some testing with the ignore GUI. Here are some of my observations.

1. This commit causes the whole Web GUI to stop working in Internet Explorer 11.

   SCRIPT1002: Syntax error
   syncthingController.js (1913,71)
   SCRIPT1010: Expected identifier
   ignoreTreeService.js (5,5)
   Error: [ng:areq] Argument 'SyncthingController' is not a function, got undefined
   http://errors.angularjs.org/1.3.20/ng/areq?p0=SyncthingController&p1=not%20a%20function%2C%20got%20undefined
      at assertArg (http://localhost:8386/vendor/angular/angular.js:1609:5)
      at assertArgFn (http://localhost:8386/vendor/angular/angular.js:1619:3)
      at Anonymous function (http://localhost:8386/vendor/angular/angular.js:8512:9)
      at Anonymous function (http://localhost:8386/vendor/angular/angular.js:7680:13)
      at forEach (http://localhost:8386/vendor/angular/angular.js:353:11)
      at nodeLinkFn (http://localhost:8386/vendor/angular/angular.js:7667:11)
      at compositeLinkFn (http://localhost:8386/vendor/angular/angular.js:7159:13)
      at publicLinkFn (http://localhost:8386/vendor/angular/angular.js:7038:30)
      at Anonymous function (http://localhost:8386/vendor/angular/angular.js:1479:11)
      at Scope.prototype.$eval (http://localhost:8386/vendor/angular/angular.js:14589:9)
   

   

2. Windows path backslashes \ seem to be completely unsupported. Trying to use them in paths automatically disables the Basic button.

   

   Just a side note, but when the button becomes disabled, it would be nice to have some kind of an indicator as to why it is unavailable. Otherwise the users will likely be confused.

3. The whole "Edit Folder" GUI becomes extremely laggy with a large number of files/folders. I have tried to use it with 10,000 files, and it is barely usable. I am testing on a decent hardware in Chromium, so the situation will likely be much worse on slower devices. CPU spikes to 100% and RAM usage goes up significantly (e.g. from 60 MB to 200 MB) upon trying to view the ignore patterns. Also, scrolling through such a long list is cumbersome in itself too. I would say that this requires pagination to be usable with large numbers of items. If pagination is impossible, then maybe it would be better to have it contained in a box with a separate scrollbar?

   Just to reiterate, the lag is present before even clicking on "Ignore Patterns" themselves. As soon as the "Edit Folder" popup opens, the whole GUI becomes laggy.

4. Numbers are sorted as follows.

   

   It would be nice to have them sorted by their numerical value instead, but I am not sure what exactly is responsible for the sorting behaviour.

5. There are some misalignment issues in the tree,

   

   but the same is true when browsing Versions, so I am not sure whether anything can be done about this here.

Awesome feature - I found this code when I was motivated to write something similar myself.
@christianprescott any plans for the two features:

• Show the "Basic" view in "readonly mode" when patterns are too complex to edit in "basic mode"?
  (I think it should be pretty easy to just show which files are included but avoid changing the check marks)
• Switch between local and global state view?
  as this is important for me to see which local files are excluded

Awesome development! I've been wishing for this functionality for years 🙂 As a stress test I synced a folder containing the Firefox source code. I expanded folders until I had more than 60.000 items in the tree. No problems (de)selecting items.

However I encountered one bug. When expanding a directory in the tree, it may use the wrong folder id when calling the rest api. You need two folders to test this. In the first folder browse the tree by expanding directories. Works fine. Then try the same on the second folder. It will fail to browse more than one level. You can see in the network requests panel that it uses the wrong folder id (id of the first folder even though we're inside the second folder now). In the screenshot you can see the current folder id at the top, and the wrong folder id used in the api call.

Besides this I'd recommend a few usability improvements.

At the moment it looks like a whole folder is included because it's checked and marked blue, even though some of its sub folders are excluded. I think it's useful to distinguish 3 states:

1. Excluded, including all children (checkbox unchecked)
2. Partly excluded, but some of its children are checked (checkbox unchecked or partially checked, maybe different background color?)
3. Fully included, including all children (checkbox checked)

The tree is under the "Ignore Patterns" tab, but what we're selecting is being included (not ignored). This is a bit contradictory in my mind. I think a heading/label above the tree could help explain, for example: "Select items to sync".

Currently this ui is not presented when accepting a shared folder. So I'd have to manually set .stignore rules to ** when accepting the folder and then go into folder settings to select folders I want to sync in the tree view. Not a big issue as I can work around it.

I'd be super happy if the current functionality would be merged. 🥳

Thanks @tomasz1986

1. Thank you for including the complete error message :) looks like I let some ES6 slip in those two places. Fixed. If there are further problems with this I'll look into adding some linting rules.
2. Changed to use the path separator from the /rest/system/status endpoint. I don't have a Windows machine on hand to test, but I would expect this change to work with Windows separators, and also across OS's? e.g. Loading the UI from a Windows server to a Linux browser.
3. Performance issues are still outstanding. Let's see what I can do about paginating client side. I suspect lots of time is spent by fancytree processing and rendering rows, not necessarily by fetching files. I will investigate ways to have fancytree do its loading in smaller batches using the existing syncthing API.
4. They're presented in the same value returned by the API. I would prefer not to make that change client side, especially if pagination becomes involved
5. Fixed here and in Versions tree. Vertically centered the icons.

@schuellerf

1. No plans. In my eyes, until all types of patterns are supported by the UI the presence of any advanced patterns in a "readonly" mode makes it misleading because it would present an inaccurate/incomplete picture of ignored files. Besides, I think a more worthwhile way to support complex patterns would be to include them in the API, with the actual golang matching engine telling the client which pattern matches a file.
2. API support for this does not exist yet. /rest/system/browse almost does - but only presents directories. It's an interesting distinction though because when a file is ignored on all devices globally it disappears! And can't be included again until the ignoring pattern is removed. So this would be a good enhancement.

@Jip-Hop
Good catch with the folder change - fixed.
Some descriptive text has been added: "Selected items will be synced, other items ignored" but we're still under the Ignore Patterns tab for now.
Using the current db/browse endpoint we can't display folder contents until the folder is present on our own syncthing device, so this isn't possible yet. However one enhancement I saw mentioned elsewhere was a toggle for top-level "ignore root" which would be pretty simple.

Thanks! I like the descriptive text.

Tested again, this time on Windows. I confirm the folder switching is now working properly 🙂

However (un)checking doesn't change the checkbox any more and duplicate ignore rules are added. See screenshots for results after clicking a checkbox multiple times.

Would indeed be nice to show a top-level "ignore root" button (similar to manually typing the ** rule). Currently Basic mode under Ignore Patterns is disabled in the Add Folder modal. I also suggest some explanation that after clicking Save it's possible to select which files/folders to sync form the Folder -> Edit -> Ignore Patterns -> Basic menu.

Have you considered tri-stage checkboxes (selectMode: 3), to not show 'fully checked' if at lease one child is excluded?

Thanks @tomasz1986

1. Thank you for including the complete error message :) looks like I let some ES6 slip in those two places. Fixed. If there are further problems with this I'll look into adding some linting rules.
2. Changed to use the path separator from the /rest/system/status endpoint. I don't have a Windows machine on hand to test, but I would expect this change to work with Windows separators, and also across OS's? e.g. Loading the UI from a Windows server to a Linux browser.
3. Performance issues are still outstanding. Let's see what I can do about paginating client side. I suspect lots of time is spent by fancytree processing and rendering rows, not necessarily by fetching files. I will investigate ways to have fancytree do its loading in smaller batches using the existing syncthing API.
4. They're presented in the same value returned by the API. I would prefer not to make that change client side, especially if pagination becomes involved
5. Fixed here and in Versions tree. Vertically centered the icons.

I have done some quick testing.

1. The GUI loads in IE 11, and it is possible to edit advanced ignore patterns, but basic ones do not load. There are still some JS errors.

   TypeError: Object doesn't support property or method 'find'
      at Anonymous function (http://localhost:8391/syncthing/core/syncthingController.js:1965:17)
      at processQueue (http://localhost:8391/vendor/angular/angular.js:13337:11)
      at Anonymous function (http://localhost:8391/vendor/angular/angular.js:13353:27)
      at Scope.prototype.$eval (http://localhost:8391/vendor/angular/angular.js:14589:9)
      at Scope.prototype.$digest (http://localhost:8391/vendor/angular/angular.js:14405:15)
      at Scope.prototype.$apply (http://localhost:8391/vendor/angular/angular.js:14694:13)
      at Anonymous function (http://localhost:8391/vendor/angular/angular.js:14984:11)
      at completeOutstandingRequest (http://localhost:8391/vendor/angular/angular.js:4959:7)
      at Anonymous function (http://localhost:8391/vendor/angular/angular.js:5347:7)
   

2. The \ separators seem to work now, but there are issues with duplicate patterns and non-working checkboxes mentioned above by @Jip-Hop, so it is kind of difficult to test right now 😉.

3. The same problems affect the Versions modal (see gui: Tweak the Restore Versions modal for better usability #7395 and Restore Versions modal extremely laggy and RAM-hungry with many items #7393). I have tried some basic HTML and CSS tweaks, but there has been no difference so far. It is just the whole loading and processing that is super heavy, both on CPU and RAM. I would say that either pagination or some kind of lazy load will be required to make it usable with larger numbers of files. If you manage to achieve this here, then I'm quite sure that the same could be applied to the Versions modal to fix those issues there too 🙂.

   Also, I have added some tweaks to contain the box and make it resizable in gui: Tweak the Restore Versions modal for better usability #7395, which probably could be adapted to both the Versions and the Ignore Patterns tree at the same time.

4. I have done the same with

   span.fancytree-expander,
   span.fancytree-icon {
       vertical-align: middle !important;
   }
   

   in gui: Tweak the Restore Versions modal for better usability #7395. Not saying which code is better, but if my commit is applied first, then you probably will not need to have any icon aligning tweaks here (and vice versa).

🤖 beep boop

I'm going to close this pull request as it has been idle for more than 90 days.

This is not a rejection, merely a removal from the list of active pull requests that is periodically reviewed by humans. The pull request can be reopened when there is new activity, and merged once any remaining issues are resolved.

I think this would be a really useful addition. @christianprescott are you still interested in finishing the implementation?

Another ping @christianprescott, since you seem to have been active on GitHub again lately. This being one of the most requested features in Syncthing, would you be willing to start working on it again? I see you switched jobs around the time this PR went quiet. Maybe now is a better time?

We also have some related progress toward a selective sync GUI using ignore patterns, see #9619 and its immediate user: https://forum.syncthing.net/t/beta-test-my-new-ios-app-for-syncthing/22457/1