* Functional specification for issue #516: Obliterate * #TODO + add the missing functional requirements + add more details and examples - where needed - on the functional requirements + add the missing non-functional requirements + probably need to split up some of the requirements to make them smaller (and SMARTer). + add a clear description of the cascading effect: revisions contain directories, directory contains files and property changes, files contains content changes and property changes, content and property changes can be merged. + verify that with the documented requirements (functional and non-functional) the use cases and examples can be 'solved'. + fill in use case vs requirements table. + let a native English speaker review for clarity and propose better keywords + review on mailing list(s). + finalize #END-TODO I. Overview This document serves as the functional specification for what's commonly called the 'svn obliterate' feature. II. Use Cases 1. Disable all access to confidential information in a repository. [security] A. Description This is the case where a user has added information to the repository that should not have been made public. The distribution of this information must be halted, and where it has been distributed, it must be removed. This use case typically requires removal of any trace of that information from the whole history of the repository. In short, if a confidential file was copied, also obliterate the copy. B. Examples + User adding documents with confidential information to the repository. Needs to stop distribution to working copies and mirrors ASAP. + User adding source code to the repository, finds out later that it's infringing certain intellectual rights. Need to remove all traces of the infringing source code, including all derivatives, from the repository. C. Primary actor triggering this use case A key user of the repository that knows what confidential information should be removed, and who can estimate the impact of obliteration (which paths, which revision range(s) etc. Normal users should not be able to obliterate. For those users we already have 'svn rm'. 2. Remove obsolete information from a repository and free the associated disc space. [disc space] A. Description This is the case where unneeded or obsolete information is stored in the repository, taking up lots of disc space. In order to free up disc space, this information may be obliterated. This use case typically requires removal of certain subsets of the repository while leaving later revisions intact. In short, if an obsolete file was copied, leave the copy intact. This use case is often combined with archiving of the obsolete information: archive first, then obliterate. B. Examples + User adding a whole set of development tools, huge binaries or external libraries to the product by mistake. + Users managing huge files (MB/GB's) as part of their normal workflow. These files can be removed when work on newer versions has started. + Users adding source code, assets and build deliverables in the same repository. Certain assets or build deliverables can be removed + When a project is moved to its own repository, the project's files may be obliterated from the original repository. This includes moving old projects to an archive repository. + Repositories setup to store product deliverables. Those deliverables for old unmaintained versions, like everything older than a revision or date, may be obliterated from the repository. + Removal of dead branches which changes have and will not be included in the main development line. C. Primary actor triggering this use case A repository administrator that's concerned about disc space usage. However, only a key user can decide which information may be obliterated. III. Current solution 1. Dump -> Filter -> Load Subversion already has a solution in place to completely remove information from a repository. It's a combination of dumping a repository to text format (svnadmin dump), using filters to remove some nodes or revisions from the text (svndumpfilter) and then loading it back into a new repository (svnadmin load). Where svndumpfilter is used to remove information from a repository, obliterate should cover at least all of its features. 2. Advantages of current solution + svndumpfilter exists today. + It has the most basic include and exclude filters built-in. + Its functionality is reasonably well understood. 3. Disadvantages of current solution + svndumpfilter has a series of issues (8 right now, see the issue tracker). + Its filtering options are limited to include or exclude paths, no wildcard support... + Filtering is based on pathnames, not node based + Due to its streamy way of working it has no random access to the source nor target repository, hence it can't rewrite copies or later modifications on filtered files. + Uses an intermediate text format and requires filtering the whole repository, not only the relevant revisions -> Slow. + Requires the extra disc space for the output repository. + The svndumpfiler code is not actively maintained. + Slow. + Requires shell access on repository server or at least access to dump files. IV. Detailed functional requirements 0. Overview The workflow of the obliterate solution can be defined in six steps: 1. SELECT the lines of history to obliterate. 2. LIMIT the range of obliteration to a revision or revision range. 3. DEFINE how to handle the consequences of obliteration on derivative modifications. [#TODO: this needs a clearer keyword] 4. HIDE the selected modifications. 5. If needed, UNHIDE selected modifications. 6. OBLITERATE the selected modifications from the repository. While in the final solution step 4 HIDE and step 5 OBLITERATE may be combined into one - as it's probably much easier to implement, there are some clear advantages to keeping the HIDE step separate: + In the security use case, hiding confidential information is much more time-critical than the final obliteration. + Hiding information can be done by a key user, whereas obliteration should be done by an administrator with direct repository access. Note: while there's certainly a need to have repository administration control without requiring shell access to a server, this need is not obliterate specific and as such doesn't have to be solved in the scope of this solution. + Hiding information can be seen as a dry run for final obliteration. It allows the key user to analyse the impact of the selected filters, hide extra information or recover where needed before committing to removing it from the repository. Each of these steps are detailed in the following list of functional requirements. We'll probably find that the differences in requirements needed for each use cases are mainly in step 3 and 4. Priorities are one of: ( MoSCoW ) + M - MUST have this. + S - SHOULD have this if at all possible. + C - COULD have this if it does not affect anything else. + W - WON'T have this time but WOULD like in the future. 1. SELECT a modification to obliterate. A. Description Allow the user to obliterate a single modification from the repository. The lowest level of modification we should consider is the change to a file or directory committed in a specific revision. (Read: no need to support obliterating a single line in a document) A modification can be selected by: + A path name + A PEG revision, default is HEAD. + A revision (FROM revision equals TO revision) This requirement can be seen as a combination of: - SELECT a file or directory. - LIMIT the range to the selected revision. B. Main use case all C. Primary actor key user D. Priority M - MUST have this. 2. SELECT a file to obliterate. A. Description Allow the user to obliterate a file from the repository. The file can be selected by: + A path name + A PEG revision, default is HEAD. If the file was copied from another file, we should have the option to select either: + the copy + the file's ancestor B. Main use case all C. Primary actor key user D. Priority M - MUST have this. 3. SELECT a directory to obliterate A. Description Allow the user to obliterate a directory, including all its children, the whole tree. The directory can be selected by: + A path name + A PEG revision, default is HEAD. If the directory was copied from another directory, we should have the option to select either: + the copy + the directory's ancestor Some of the children of the directory might be 'older' than the directory itself. This normally happens when the directory was copied from another directory (branched, tagged). B. Main use case all C. Primary actor key user D. Priority M - MUST have this. 4. SELECT all modifications in a revision to obliterate A. Description Allows the user to obliterate all modifications made in: + A revision (FROM revision equals TO revision) This is equal to: - SELECT the root of the repository. - LIMIT the range to the selected revision. It should be possible to choose whether or not to obliterate: + the log message, author and date properties + all other revision properties. Obliterating the HEAD revision can be seen as a special case of this requirement. Note: the revision number itself does not need to removed. B. Main use case all C. Primary actor key user D. Priority SHOULD have this if at all possible. 5. SELECT multiple modifications, files or directories to obliterate A. Description Allows the user to obliterate multiple modifications, files or directories. Modifications can be selected by: + A list of PATH@PEGREV's + revisions Paths can be selected by: + A list of path@PEGREV's. + Wildcards: '*.jpg', 'build_*' B. Main use case all C. Primary actor key user D. Priority SHOULD have this if at all possible. 6. LIMIT the range between FROM and TO revisions or dates. A. Description Both FROM and TO may be specified in the form of revisions, dates or keywords like HEAD. This is the most general case, where both FROM revision and TO revision can be specified. Depending on which SELECT option was chosen, the default LIMITs will be different, as detailed in this table: +--------------+---------------------------+---------------+ | SELECT | LIMIT FROM rev | LIMIT TO rev | +--------------+---------------------------+---------------+ | modification | HEAD | HEAD | | file | creation rev | HEAD | | directory | creation rev | HEAD | | \ children | creation rev of directory | HEAD | +--------------+---------------------------+---------------+ B. Main use case all C. Primary actor key user D. Priority M - MUST have this. 7. LIMIT the range between PATH CREATION and TO revisions or dates. A. Description This LIMIT can only be used when SELECTing files or directories, not with modifications. This is a special case of requirement IV.6., where the FROM revision is defined as the revision in which the selected file or directory was either: - created - copied from another file or directory Implementation Note: Can this be implemented through a new keyword PATH-CREATION and PATH-LAST-COPY revision? This doesn't need to be obliterate specific. B. Main use case all C. Primary actor key user D. Priority M - MUST have this. E. Workaround As it's difficult right now to make the distinction between a copy of a directory and a rename, and a directory might be renamed a few times after it was copied, we might need to use a PEG revision to indicate where the real directory copy revision can be found. 9. DEFINE: Include all descendants in the obliteration of a file A. Description This is basically a greedy obliteration, where all places in the repository where a file or a modification to a file has propagated through copies or later modifications is also obliterated. When obliterating a file, the impact of this obliteration should be checked in the selected revision range in the repository. Depending on the type of modification, actions should be taken. When the file is: + Added: This is the creation point of the file. Remove the Add operation and the content and properties delta. + Deleted: Remove the Delete operation. + Replaced by TARGET: see Deleted. Will become Copy operation of the TARGET. + Copied to TARGET (or resurected): delete the Copied operation and drop copy-from path and rev. Add the TARGET file in the selection of to be obliterated files, using the same limit (revision range) and impact-on-descendants option. + Moved to TARGET: delete the Copy+Delete operations and drop copy-from path and rev. Add the TARGET file in the selection of to be obliterated files, using the same limit (revision range) and impact-on-descendants option. + Modified: delete the Modified operation and the delta. Add the modification (file-revision) in the selection of to be obliterated modifications, using the same limit (revision range) and impact-on-descendants option. #TODO: what to do when the TO revision is older than HEAD. B. Main use case security C. Primary actor key user D. Priority M - MUST have this 9. DEFINE: Exclude all descendants from the obliteration of a file A. Description If the obliterated information is still needed in a later revision in the repository, the information will be restored in that later revision. When obliterating a file, the impact of this obliteration should be checked in the selected revision range in the repository. Depending on the type of modification, actions should be taken. When the file is: + Added: This is the creation point of the file. Remove the Add operation and the content and properties delta. + Deleted: when the file is obliterated earlier, there's nothing to Delete anymore. Remove the Delete operation. + Replaced by TARGET: see Deleted. Will become Copy operation of the replacing file. + Copied to TARGET (or resurected): replace the Copy operation with Add (drop copy-from path and rev), find the original contents and properties of the file at the copy-from revision and use these for the new TARGET. #TODO: what to do when the Copy was modified in the working copy before committing. #END-TODO + Moved to TARGET: is combination of Deleted and Copied. Will become Add of the TARGET with the original content and properties. + Modified: replace the Modified operation with Add, find the original content and properties of the ancestor, apply the delta to that content and properties and use the result to recreate the file. Note: only the first change after the obliterated revision of the file should be handled, except for copies of the now obliterated revision. Example: r1: A iota "original content\n" r2: M iota "original content\nextra line\n" r3: D iota r4: A cp-iota (copy from iota@1) "original content\n" Here we obliterate iota, range -r 1:1, exclude descendants. Result: So, r1 will be obliterated, r2 will be rewritten, r3 should be ignored. Since r4 is based on the now obliterated r1, it should be rewritten as 'A cp-iota' with the content and properties of iota@1. r1: [obliterated] r2: A iota "original content\nextra line\n" r3: D iota r4: A cp-iota "original content\n" Note for implementation: if at all possible, this should be implemented so that we don't need more copies of the information than before the obliteration, to avoid increasing the repository size. If not possible, this requirement will only make sense for files that have never changed or copied. B. Main use case disc space C. Primary actor key user D. Priority M - MUST have this. 10. DEFINE: Include all descendants in the obliteration of a directory A. Description B. Main use case security C. Primary actor key user D. Priority M - MUST have this 11. DEFINE: Exclude all descendants from the obliteration of a directory A. Description When obliterating a directory, the impact of this obliteration should be checked in the selected revision range in the repository. Depending on the type of modification, actions should be taken. When the file is: #TODO: add effects of directory operations B. Main use case disc space C. Primary actor key user D. Priority M - MUST have this. 12. DEFINE: Include all descendants in the obliteration of a modification A. Description Now that Subversion 1.5 includes merge tracking we have the option to find out how modifications cascade through the repository with merge operations. #IMPL-DETAIL A merge operation is identified by a change of type Modification that includes a change to the svn:mergeinfo property. #END-OF-IMPL-DETAIL When obliterating a modification, the impact of this obliteration should be checked in the selected revision range in the repository. Depending on the type of modification, actions should be taken. When the modification is: + Deleted: + Replaced by TARGET: + Copied to TARGET: + Moved to TARGET: + Merged to TARGET: + Modified: + Merged to TARGET: #TODO: define how to select descendants. B. Main use case security C. Primary actor key user D. Priority C - COULD have this if it does not affect anything else. 13. DEFINE: Exclude all descendants from the obliteration of a modification A. Description If the obliterated modification is still needed in a later revision in the repository, that modification will be made available in that later revision. When obliterating a modification, the impact of this obliteration should be checked in the selected revision range in the repository. Depending on the type of modification, actions should be taken. + Deleted: Can be ignored. + Replaced by TARGET: Can be ignored. + Copied to TARGET: + Moved to TARGET: + Merged to TARGET: the modification is merged to another file. This action can be ignored because when merging a delta to another path, that delta is copied and reapplied to the new path, not relying on the content of the original delta. #TODO: is that true for both FSFS and BDB? Are we not relying on an implementation detail that can change in the future? + Modified: If the modification contains lines that were modified or added in the now obliterated delta, find the original content and properties of the ancestor, apply the delta to that content and properties and use the result to recreate the file. #TODO: define how to select descendants. B. Main use case disc space C. Primary actor key user D. Priority M - MUST have this. 14. LOG selected modifications, files, directories and revisions A. Description This is essentially a dry run of the obliterate action. In order to assess the impact of the selected filters, the user wants to see the list of to-be obliterated paths first. The result should be printed to the console and contain the info as shown in this example: +-----------------------------------------------+-------------------+ | Revision | Current action | Path | New action | | | | | after obliteration| +----------+----------------+-------------------+-------------------+ | r100 | A | /trunk/SECRET | [obliterated] | | r101 | M | /trunk/SECRET | [obliterated] | | r105 | A+ | /branch/1.0/SECRET| A | +----------+----------------+-------------------+-------------------+ B. Main use case all C. Primary actor key user D. Priority M - MUST have this. 15. HIDE selected files, directories and revisions A. Description #TODO B. Main use case all C. Primary actor key user D. Priority S - SHOULD have this if at all possible. 16. UNHIDE selected files, directories and revisions A. Description #TODO B. Main use case all C. Primary actor key user D. Priority S - SHOULD have this if at all possible. 17. OBLITERATE selected modifications, files, directories and revisions A. Description #TODO B. Main use case all C. Primary actor administrator D. Priority M - MUST have this. 18. Keep audit trail of obliterated information A. Description #TODO B. Main use case security C. Primary actor administrator D. Priority M - MUST have this. 19. Propagating obliteration info to working copies A. Description #TODO B. Main use case all C. Primary actor administrator D. Priority C - COULD have this if it does not affect anything else. E. Workaround 20. Propagating obliteration info to mirrors A. Description #TODO B. Main use case security C. Primary actor administrator D. Priority C - COULD have this if it does not affect anything else. E. Workaround [..] V. Detailed non-functional requirements 1. Authorization for hiding the information 2. Authorization for restoring the information 3. Authorization for obliterating information from the repository 4. Limit repository downtime 5. Maintain integrity of the repository 6. Limit temporary disc space 7. Compatibility with older Subversion clients [..] VI. Requirements vs Use Cases This table matches the requirements with the use cases. It tries to answer two specific questions: 1. What's the value of a requirement in terms of the use cases? 2. Which requirements do we need to implement to really solve a specific use case. +-------------------------------------------------------------------------+ | Disable all access to confidential information in a repository --- | | Remove obsolete information from a repository | | \ | +----------------------------------------------------------+------+-------+ + [ #TODO: fill in when all reqs are defined ] | x | x | + | | x | +-------------------------------------------------------------------------+ VII. Appendix 1. Link to external documentation [1] Issue 516: http://subversion.tigris.org/issues/show_bug.cgi?id=516 [2] Karl Fogel's proposal to use the replay API and filters: http://svn.haxx.se/dev/archive-2008-04/0687.shtml [3] Bob Jenkins's thread about "Auditability": keep log of what has been obliterated: http://svn.haxx.se/dev/archive-2008-04/0816.shtml [4] Users discussing some examples of the need for obliterate: http://svn.haxx.se/users/archive-2005-04/0715.shtml [The corresponding technical specification will be put in another document]