Gradual fetching of data by scrolling down in File Manger Application

[mahsashadi]

According to this issue, I am implementing gradual fetching by scrolling in FM app. I have two issues.

1. I need to detect when the whole data is fetched to stop fetching the next time scroll hits the bottom.
   So I think I need to initially get total number of directories and files of a path (without calling readdir api and getting the list) and save it into some state.
   I checked the stat api, but it does not provide this attribute.
2. I need to have the last file returned from server, but the list returned here is not in the same order as server response, due to some sorting done here.

[mahsashadi]

BTW, What is your idea about adding this pagination feature to your file manager app?
As you suggest here, it can be used by adapters supported this interface in their api.
If you agree, I can make a pull request when it is totally done.

[andersevenrud]

Hm.

Relying on either the total number of files or the last one to figure out when to stop does not sound like a good idea.

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

[mahsashadi]

An easier way to do this is simply to set a "page size" and whenever you get a "page" back from the api that is less than this page size or zero, you know you've reached the end.

Yes, seems a better idea, thanks :)

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

[andersevenrud]

but I still need to have the last file of each page, returned by server, as a token for getting the next page.

It is possible just to use a page number 🤔

const pageSize = 10
let currentPage = 0
let reachedEnd = false

function changedDirectory() {
  currentPage = 0
  reachedEnd = false
}

async function scrollReachedBottom() {
  if (reachedEnd) return
  const nextPage = currentPage + 1
  const result = await readdir('home:/', { page: nextPage, pageSize })
  reachedEnd = result.length < pageSize
  currentPage = nextPage
}

Now you have a offset on the server: page * pageSize.

[mahsashadi]

But it seems my storage pagination API does not support offset.
https://docs.openstack.org/swift/latest/api/pagination.html

[andersevenrud]

I see. I'm not really sure how to solve this then without also adding sorting as an option (default true).

[mahsashadi]

So do you think you can add this feature?
Or if it is needed to be done by myself, I can start working on that.

[andersevenrud]

Feel free to have a look at it. This would also mean that the file manager would have to run the sort manually to make it behave the same.

[mahsashadi]

Great :)
Do you have any idea which parts of codes are potential for changing and also where this boolean option should be set.

[andersevenrud]

I was just thinking of adding it to the options and add the logic to the VFS function:

osjs-client/src/vfs.js

Line 84 in 385b38f

export const readdir = (adapter, mount) => (path, options = {}) =>

osjs-client/src/vfs.js

Lines 70 to 75 in 385b38f

	const handleDirectoryList = (path, options) => result =>
	Promise.resolve(result.map(stat => createFileIter(stat)))
	.then(result => transformReaddir(pathToObject(path), result, {
	showHiddenFiles: options.showHiddenFiles !== false,
	filter: options.filter
	}));

[mahsashadi]

This would also mean that the file manager would have to run the sort manually to make it behave the same.

About this part, when a new page is fetched, I think sorting the accumulative list (previous pages+ new page) leads to a bad UX. The files of previous pages and the new one might be mixed, and the user can not scroll just among files of new page.

[andersevenrud]

I was only thinking about applying sorting per result.

Because in adapters that does not support this you'd only get one result.

[mahsashadi]

So by adding sortFiles option to this method as below:

// Handles directory listing result(s)
const handleDirectoryList = (path, options) => result =>
  Promise.resolve(result.map(stat => createFileIter(stat)))
    .then(result => transformReaddir(pathToObject(path), result, {
      showHiddenFiles: options.showHiddenFiles !== false,
      filter: options.filter,
      sortFiles: options.sortFiles
    }));

We just need to check sortFiles option in transformReaddir method and do sorting if it is true. This way it works :)

The question is where should I config this boolean option? Maybe it is needed to add this config to src/client/config.js:

  fileManager:{
    pageSize: 1000,
    sortFiles: false
  }

[andersevenrud]

The question is where should I config this boolean option?

Doesn't this just fit right into the FM readdir function ?

[mahsashadi]

BTW, What is your idea about adding this pagination feature to your file manager app?
As you suggest here, it can be used by adapters supported this interface in their api.
If you agree, I can make a pull request when it is totally done.

So when we complete this feature, are you going to add it to your FM app? If so is it possible to set in readdir as below?

const options = {
  showHiddenFiles: proc.settings.showHiddenFiles,
  sortFiles: false,
  page:{
    size:1000,
    number: state.currentPage,
    lastFetchedFile: state.lastFetchedFile
  }
};

In this way when sorting (deafult is true) is going to happen?