fsmonitor: don't fill bitmap with entries to be removed #372

gitgitgadget · 2019-10-03T23:39:31Z

On the Git mailing list, Junio C Hamano wrote (reply to this):

"William Baker via GitGitGadget" <[email protected]> writes: > create mode 100755 t/t7519/fsmonitor-env > ... > + if (pos >= istate->cache_nr) > + BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")", > + (uintmax_t)pos, (uintmax_t)istate->cache_nr); This is how we show size_t values without using "%z" that we avoid, but are "pos" and 'cache_nr" size_t or ssize_t? I thought they are plain boring unsigned, so shouldn't we use the plain boring "%u" without casting? The same comment applies to other uses of uintmax_t cast in this patch. > void fill_fsmonitor_bitmap(struct index_state *istate) > { > - unsigned int i; > + unsigned int i, skipped = 0; > istate->fsmonitor_dirty = ewah_new(); > - for (i = 0; i < istate->cache_nr; i++) > - if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID)) > - ewah_set(istate->fsmonitor_dirty, i); > + for (i = 0; i < istate->cache_nr; i++) { > + if (istate->cache[i]->ce_flags & CE_REMOVE) > + skipped++; > + else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID)) > + ewah_set(istate->fsmonitor_dirty, i - skipped); > + } > } Matches the explanation in the proposed log message pretty well. Good job. > @@ -354,4 +354,16 @@ test_expect_success 'discard_index() also discards fsmonitor info' ' > test_cmp expect actual > ' > > +# Use test files that start with 'z' so that the entries being added > +# and removed appear at the end of the index. In other words, future developers are warned against adding entries to and leaving them in the index that sort later than z100 in new tests they add before this point. Is the above wording clear enough to tell them that, I wonder? > +test_expect_success 'status succeeds after staging/unstaging ' ' > + test_commit initial && > + removed=$(test_seq 1 100 | sed "s/^/z/") && Thanks.

On the Git mailing list, William Baker wrote (reply to this):

On 10/3/19 4:36 PM, Junio C Hamano wrote: >> + if (pos >= istate->cache_nr) >> + BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %"PRIuMAX")", >> + (uintmax_t)pos, (uintmax_t)istate->cache_nr); > > This is how we show size_t values without using "%z" that we avoid, > but are "pos" and 'cache_nr" size_t or ssize_t? I thought they are > plain boring unsigned, so shouldn't we use the plain boring "%u" > without casting? > > The same comment applies to other uses of uintmax_t cast in this > patch. > Thanks for catching this. I will update these BUGs in the next patch to avoid casting. >> +# Use test files that start with 'z' so that the entries being added >> +# and removed appear at the end of the index. > > In other words, future developers are warned against adding entries > to and leaving them in the index that sort later than z100 in new > tests they add before this point. Is the above wording clear enough > to tell them that, I wonder? > You're understanding is correct, and I agree this comment could be clearer. I will fix this up in v2. Thanks for the feedback! William

gitgitgadget · 2019-10-10T11:10:58Z

On the Git mailing list, SZEDER Gábor wrote (reply to this):

On Wed, Oct 09, 2019 at 02:00:12PM -0700, William Baker via GitGitGadget wrote: > diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh > index 81a375fa0f..87042470ab 100755 > --- a/t/t7519-status-fsmonitor.sh > +++ b/t/t7519-status-fsmonitor.sh > @@ -354,4 +354,17 @@ test_expect_success 'discard_index() also discards fsmonitor info' ' > test_cmp expect actual > ' > > +# This test covers staging/unstaging files that appear at the end of the index. > +# Test files with names beginning with 'z' are used under the assumption that > +# earlier tests do not add/leave index entries that sort below them. > +test_expect_success 'status succeeds after staging/unstaging ' ' > + test_commit initial && This is confusing: this is the 29th test case in this script and it creates an "initial" commit?! The first "setup" test case has already created an initial commit, so this should rather be called "second". OTOH, none of the later commands in this test case seem to have anything to do with this second commit, and indeed the test case works even without it (i.e. 'git status' still segfaults without the fix and then succeeds with the fix applied), so instead of updating its message perhaps it could simply be removed. > + removed=$(test_seq 1 100 | sed "s/^/z/") && > + touch $removed && > + git add $removed && > + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" && > + FSMONITOR_LIST="$removed" git restore -S $removed && > + FSMONITOR_LIST="$removed" git status > +' > + > test_done

On the Git mailing list, SZEDER Gábor wrote (reply to this):

On Thu, Oct 10, 2019 at 01:07:32PM +0200, SZEDER Gábor wrote: > On Wed, Oct 09, 2019 at 02:00:12PM -0700, William Baker via GitGitGadget wrote: > > diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh > > index 81a375fa0f..87042470ab 100755 > > --- a/t/t7519-status-fsmonitor.sh > > +++ b/t/t7519-status-fsmonitor.sh > > @@ -354,4 +354,17 @@ test_expect_success 'discard_index() also discards fsmonitor info' ' > > test_cmp expect actual > > ' > > > > +# This test covers staging/unstaging files that appear at the end of the index. > > +# Test files with names beginning with 'z' are used under the assumption that > > +# earlier tests do not add/leave index entries that sort below them. I just read through Junio's comments on the first version of this patch, in particular his remarks about this comment. If this new test case below were run in a dedicated repository, then this comment wouldn't be necessary, and all my comments below about that not-really-initial commit would be moot, too. > > +test_expect_success 'status succeeds after staging/unstaging ' ' > > + test_commit initial && > > This is confusing: this is the 29th test case in this script and it > creates an "initial" commit?! > > The first "setup" test case has already created an initial commit, so > this should rather be called "second". > > OTOH, none of the later commands in this test case seem to have > anything to do with this second commit, and indeed the test case works > even without it (i.e. 'git status' still segfaults without the fix and > then succeeds with the fix applied), so instead of updating its > message perhaps it could simply be removed. > > > + removed=$(test_seq 1 100 | sed "s/^/z/") && > > + touch $removed && > > + git add $removed && > > + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" && > > + FSMONITOR_LIST="$removed" git restore -S $removed && > > + FSMONITOR_LIST="$removed" git status > > +' > > + > > test_done

On the Git mailing list, William Baker wrote (reply to this):

On 10/10/19 4:22 AM, SZEDER Gábor wrote: >>> +# This test covers staging/unstaging files that appear at the end of the index. >>> +# Test files with names beginning with 'z' are used under the assumption that >>> +# earlier tests do not add/leave index entries that sort below them. > > I just read through Junio's comments on the first version of this > patch, in particular his remarks about this comment. > > If this new test case below were run in a dedicated repository, then > this comment wouldn't be necessary, and all my comments below about > that not-really-initial commit would be moot, too. > Thanks for this suggestion! I will submit a v3 version of the patch with an update to the test script. - William

gitgitgadget · 2019-10-12T01:27:50Z

On the Git mailing list, Junio C Hamano wrote (reply to this):

"William Baker via GitGitGadget" <[email protected]> writes: > +# Test staging/unstaging files that appear at the end of the index. Test > +# file names begin with 'z' so that they are sorted to the end of the index. Well, the test is now done in a freshly created repository, so the z* files are the only thing you have in here---technically they are at the end of the index, but so they are at the beginning, too. Would it affect the effectiveness of the test that you do not have any other paths in the working tree or in the index, unlike the test in the previous rounds that did not use a newly created test repository? This is not a rhetorical question, but purely asking. "no, this still tests what we want to test and shows breakage when the fix to the code in the patch gets reverted" is perfectly a good answer, but in that case, is "the end of" the most important trait of the condition this test is checking? Wouldn't the bug be exposed as long as we remove sufficiently large number of entries (like "removing more paths than the paths still in the index at the end" or something like that)? Thanks. > +test_expect_success 'status succeeds after staging/unstaging ' ' > + test_create_repo fsmonitor-stage-unstage && > + ( > + cd fsmonitor-stage-unstage && > + test_commit initial && > + git update-index --fsmonitor && > + removed=$(test_seq 1 100 | sed "s/^/z/") && > + touch $removed && > + git add $removed && > + git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" && > + FSMONITOR_LIST="$removed" git restore -S $removed && > + FSMONITOR_LIST="$removed" git status > + ) > +' > + > test_done > diff --git a/t/t7519/fsmonitor-env b/t/t7519/fsmonitor-env > new file mode 100755 > index 0000000000..8f1f7ab164 > --- /dev/null > +++ b/t/t7519/fsmonitor-env > @@ -0,0 +1,24 @@ > +#!/bin/sh > +# > +# An test hook script to integrate with git to test fsmonitor. > +# > +# The hook is passed a version (currently 1) and a time in nanoseconds > +# formatted as a string and outputs to stdout all files that have been > +# modified since the given time. Paths must be relative to the root of > +# the working tree and separated by a single NUL. > +# > +#echo "$0 $*" >&2 > + > +if test "$#" -ne 2 > +then > + echo "$0: exactly 2 arguments expected" >&2 > + exit 2 > +fi > + > +if test "$1" != 1 > +then > + echo "Unsupported core.fsmonitor hook version." >&2 > + exit 1 > +fi > + > +printf '%s\n' $FSMONITOR_LIST

On the Git mailing list, William Baker wrote (reply to this):

On 10/11/19 6:26 PM, Junio C Hamano wrote: > "William Baker via GitGitGadget" <[email protected]> writes: > >> +# Test staging/unstaging files that appear at the end of the index. Test >> +# file names begin with 'z' so that they are sorted to the end of the index. > > Well, the test is now done in a freshly created repository, so the > z* files are the only thing you have in here---technically they are > at the end of the index, but so they are at the beginning, too. > There is one other file in the index created by 'test_commit', however, the point still stands that there are almost no other entries in the index now that the test is using its own repository. > Would it affect the effectiveness of the test that you do not have > any other paths in the working tree or in the index, unlike the test > in the previous rounds that did not use a newly created test > repository? The test still validates the scenario that we're concerned about, namely that the new index that's written has less entries than the index of the last entry in the old index that's is not flagged with CE_FSMONITOR_VALID but is flagged for removal (CE_REMOVE). > This is not a rhetorical question, but purely asking. "no, this > still tests what we want to test and shows breakage when the fix to > the code in the patch gets reverted" is perfectly a good answer, but > in that case, is "the end of" the most important trait of the > condition this test is checking? Wouldn't the bug be exposed as > long as we remove sufficiently large number of entries (like > "removing more paths than the paths still in the index at the end" > or something like that)? This is exactly right. The most important trait is that the last entry flagged with CE_REMOVE does not have CE_FSMONITOR_VALID set and has an index >= the number of entries in the new index being written. I will send out a patch on top of 'wb/fsmonitor-bitmap-fix' with an update to the comment for this test. Thanks, William

wilbaker · 2019-10-11T19:26:59Z

Interestingly, without this line:

git update-index --fsmonitor &&

This test would only catch the bug when run with --run=29. If the entire script were run the issue did not repro (event with the new BUG statements in this patch).

-Original file line number
+Diff line change
@@ Expand Up @@
     static void fsmonitor_ewah_callback(size_t pos, void *is)
     {
     	struct index_state *istate = (struct index_state *)is;
-    	struct cache_entry *ce = istate->cache[pos];
+    	struct cache_entry *ce;
+    	if (pos >= istate->cache_nr)
+    		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" >= %u)",
+    		    (uintmax_t)pos, istate->cache_nr);
+    	ce = istate->cache[pos];
     	ce->ce_flags &= ~CE_FSMONITOR_VALID;
     }
@@ Expand Down Expand Up @@
     	}
     	istate->fsmonitor_dirty = fsmonitor_dirty;
+    	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+    		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+    		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     	trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful");
     	return 0;
     }
     void fill_fsmonitor_bitmap(struct index_state *istate)
     {
-    	unsigned int i;
+    	unsigned int i, skipped = 0;
     	istate->fsmonitor_dirty = ewah_new();
-    	for (i = 0; i < istate->cache_nr; i++)
-    		if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
-    			ewah_set(istate->fsmonitor_dirty, i);
+    	for (i = 0; i < istate->cache_nr; i++) {
+    		if (istate->cache[i]->ce_flags & CE_REMOVE)
+    			skipped++;
+    		else if (!(istate->cache[i]->ce_flags & CE_FSMONITOR_VALID))
+    			ewah_set(istate->fsmonitor_dirty, i - skipped);
+    	}
     }
     void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate)
@@ Expand All @@
     	uint32_t ewah_size = 0;
     	int fixup = 0;
+    	if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+    		BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+    		    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     	put_be32(&hdr_version, INDEX_EXTENSION_VERSION);
     	strbuf_add(sb, &hdr_version, sizeof(uint32_t));
@@ Expand Down Expand Up / @@ -236,6 +252,9 @@ void tweak_fsmonitor(struct index_state *istate) @@
     			}
     			/* Mark all previously saved entries as dirty */
+    			if (istate->fsmonitor_dirty->bit_size > istate->cache_nr)
+    				BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)",
+    				    (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr);
     			ewah_each_bit(istate->fsmonitor_dirty, fsmonitor_ewah_callback, istate);
     			/* Now mark the untracked cache for fsmonitor usage */
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fsmonitor: don't fill bitmap with entries to be removed #372

Uh oh!

Diff view

Diff view

There are no files selected for viewing

gitgitgadget bot Oct 3, 2019

Uh oh!

gitgitgadget bot Oct 7, 2019

Uh oh!

gitgitgadget bot Oct 10, 2019

Uh oh!

gitgitgadget bot Oct 10, 2019

Uh oh!

gitgitgadget bot Oct 11, 2019

Uh oh!

gitgitgadget bot Oct 12, 2019

Uh oh!

gitgitgadget bot Oct 15, 2019

Uh oh!

wilbaker Oct 11, 2019

Uh oh!

-Original file line number
+Diff line change
@@ Expand Up @@
     	test_cmp expect actual
     '
+    # Test staging/unstaging files that appear at the end of the index.  Test
+    # file names begin with 'z' so that they are sorted to the end of the index.
+    test_expect_success 'status succeeds after staging/unstaging ' '
+    	test_create_repo fsmonitor-stage-unstage &&
+    	(
+    		cd fsmonitor-stage-unstage &&
+    		test_commit initial &&
+    		git update-index --fsmonitor &&
+    		removed=$(test_seq 1 100 | sed "s/^/z/") &&
+    		touch $removed &&
+    		git add $removed &&
+    		git config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-env" &&
+    		FSMONITOR_LIST="$removed" git restore -S $removed &&
+    		FSMONITOR_LIST="$removed" git status
+    	)
+    '
     test_done

-Original file line number
+Diff line change
@@ -0,0 +1,24 @@
+    #!/bin/sh
+    #
+    # An test hook script to integrate with git to test fsmonitor.
+    #
+    # The hook is passed a version (currently 1) and a time in nanoseconds
+    # formatted as a string and outputs to stdout all files that have been
+    # modified since the given time. Paths must be relative to the root of
+    # the working tree and separated by a single NUL.
+    #
+    #echo "$0 $*" >&2
+    if test "$#" -ne 2
+    then
+    	echo "$0: exactly 2 arguments expected" >&2
+    	exit 2
+    fi
+    if test "$1" != 1
+    then
+    	echo "Unsupported core.fsmonitor hook version." >&2
+    	exit 1
+    fi
+    printf '%s\n' $FSMONITOR_LIST

fsmonitor: don't fill bitmap with entries to be removed #372

Uh oh!

fsmonitor: don't fill bitmap with entries to be removed #372

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

gitgitgadget bot Oct 3, 2019

Choose a reason for hiding this comment

Uh oh!

gitgitgadget bot Oct 7, 2019

Choose a reason for hiding this comment

Uh oh!

gitgitgadget bot Oct 10, 2019

Choose a reason for hiding this comment

Uh oh!

gitgitgadget bot Oct 10, 2019

Choose a reason for hiding this comment

Uh oh!

gitgitgadget bot Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

gitgitgadget bot Oct 12, 2019

Choose a reason for hiding this comment

Uh oh!

gitgitgadget bot Oct 15, 2019

Choose a reason for hiding this comment

Uh oh!

wilbaker Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!