RFC: start(arr) vs. 1 for bounds checking, iteration vs. indexing

[ScottPJones]

In the review of my PR #11575, @tkelman requested I change a 1 to start(dat) in the bounds checking.
I did find a case where start() on an AbstractArray returns a tuple, instead of 1.
I also found numerous cases, just in Base (I'm sure there are many more in the registered Packages),
where 1 was used instead of start() for bounds checking.

@nalimilan and @mbauman both responded, with some good information, and some ideas for what might need to be done to resolve these issues.

What really needs to be done here?

[yuyichao]

Shouldn't it be first?

[mbauman]

first returns the first element; this is about getting the initial index or the start of the iterable state. Those two things can be the same, but they don't have to be. I definitely don't think we should assume that they are the same.

So, is there a need for another verb here? One motivating case would be Fortran-like offset arrays, which are more possible these days without colon lowering to 1:endof(A). Unfortunately there will almost certainly be all sorts of confusion about which first is the start. Maybe first(eachindex(A)) will inline enough to be optimized away?

[mbauman]

Just getting the first index isn't good enough for an offset array. You need the first index in a given dimension. Not only is the lower bound wrong, the upper bound (size(A, d)) would be wrong, too. This may not be something that's worth bending over backwards to support, especially since the package can provide its own checkbounds method. Perhaps the answer is to just use checkbounds where-ever it's possible.

[mikewl]

Slightly tangential but couldn't start be introduced as a keyword similar to end in the array? Then A[start:end] would work for any offset array. <Moan at me to file an issue for it if anyone thinks this would be useful please!>

[mbauman]

No, end only works since it is already a reserved keyword that can't be used as a variable name.

end lowers to size or trailingsize, so it would be wrong for an offset array, too. That's actually a place where we would need this concept in Base to support such a package.

[ScottPJones]

Ufff... looks like I've opened another can of worms!
So, will the test: start(dat) <= startpos <= endof(dat) actually work for any dat::AbstractArray,
where start(dat) might return a tuple? Thanks!

[mbauman]

No. Use checkbounds(dat, startpos).

[mikewl]

Thanks! So more work would be needed in Base for offset arrays then before one could attempt such a package? Could you think of any more places work would be needed offhand?

[ScottPJones]

@Mike43110 I found just by grep'ing over a hundred places with BoundsError that would need to be examined...
@mbauman OK, thanks! 😢 Yet another change to #11575 (and I don't even use the "safe" bounds-checking version!)

[ScottPJones]

@mbauman The code has:

    startpos < start(dat) && throw(BoundsError(dat, startpos))
    (startpos <= endpos <= endof(dat)) || throw(BoundsError(dat, endpos))

checkbounds(dat,startpos) only checks the start position (and, if the end position is also checked,
unnecessarily checks it against the end of the array.
How do I also check that startpos <= endpos? It seems like there needs to be a:
checkbounds(dat,startpos,endpos) to correctly handle what my code currently does...
Help! 😀

[mbauman]

Ah, yes, that makes sense. You could use checkbounds(dat, startpos:endpos), but I wouldn't worry about it too much. Right now we make the implicit assumption that valid indices in an abstract array A along dimension d are in 1:sizeof(A, d). To do anything else would be a large change that wouldn't be your problem.

[nalimilan]

@ScottPJones Your second line doesn't look completely correct actually. If one passes endpos = startpos - 1, he will get a BoundsError about trying to index the array at an incorrect position. But that's misleading, as the problem is with startpos relative to endpos. I think you should use two calls to checkbounds, and one custom check for startpos <= endpos.

[ScottPJones]

@nalimilan OK, but will startpos <= endpos work for the tuples returned by some types of AbstractArray? What sort of error would you recommend reporting then?

[mbauman]

Indexing and iteration are separate concepts. Whatever gets returned by start (like those tuples) is a "private" iteration state and should just get passed along to next and done. You generally shouldn't index into a collection with its iteration state. So you shouldn't be passing (or receiving) those tuples as your start and end indices. If you'll be indexing with it, just use 1 as your default startpos.