API Issues with initial sync (RateLimitExceeded)

[wokawoka]

Expected vs. actual behavior

After installation and having granted a fresh token I expect to be able able to download and sync my evernote database (around 17k notes) locally on nixnote2.

Unfortunately the sync get interrupted for an API error.
I installed nixnote2 on this machine just yesterday, the token is from yesterdays, to my knowledge the API limits should be initially lifted, instead this is what I find in the log:

ERROR 2024-03-27 16:10:56.596 src/communication/communicationmanager.cpp:546 Exception stacktrace:
ERROR 2024-03-27 16:10:56.601 src/communication/communicationerror.cpp:88 "RateLimitExceeded[code=19]: API rate limit exceeded. Enable 'Sync' > 'Restart sync on API limit' or try again in 53 minutes ## >
ERROR 2024-03-27 16:10:56.601 src/threads/syncrunner.cpp:278 Error retrieving chunk
ERROR 2024-03-27 16:12:57.723 src/communication/communicationmanager.cpp:546 Exception stacktrace:
ERROR 2024-03-27 16:12:57.726 src/communication/communicationerror.cpp:88 "RateLimitExceeded[code=19]: API rate limit exceeded. Application will continue to sync in 48 minutes ## EDAMSystemException: EDAMErrorCode::RATE_LIMIT_REACHED rateLimitDuration= 2868 sec. # rateLimitDuration=\u0B34"

Restart sync on API limit (experimental) is flagged in my preferences.

Do you have any suggestion?
Thank you so much!

Specifications

• Version: 2023100407162.1.10ubuntu22.04.1

[boo-yee]

This limit is from server side. I am afraid we have to wait until the time is out.

[wokawoka]

Alright, but I have more than 20k notes to download locally in order to finish the sync
I had to restart the client at least 20 times in order to be able to download 1k of them (at the beginning I was not receiving the ratelimitexceed error above but the synchronization was getting stuck after about 10 minutes)
It would be next to impossible to complete the first sync with these limits.

I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues?
Do you guys have any suggestion about that?
Thank you very much again!

[boo-yee]

I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues?

Do you mean this link? https://dev.evernote.com/doc/articles/rate_limits.php
This page says that the rate limit of initial sync for a new token can be temporarily increased within 24 hours only, but it may be not large enough for your case, I think. This may be the reason.

If you still have Evernote installed, and your Evernote has most of your notes synced, you can transfer locally.

If your notes consist of text mainly, and not too many multimedia files, you could consider to sync with Evernote and then transfer locally. I remember that Evernote client has a limit for data amount every month for free user, I am not sure if it is enough for your notes.

I don't have so many notes, and I don't know whether the option of 'Restart sync on API limit (experimental)' works or not in reality. But if nixnote doesn't restart the sync automatically, what about restarting nixnote every one hour with an outside shell commands or something like that, with the option of auto 'sync on start up' enabled? This method may take a long time, but it is available.

[wokawoka]

I have read that with a fresh token the API limits should be initially lifted, then why am I facing all this issues?

Do you mean this link? https://dev.evernote.com/doc/articles/rate_limits.php This page says that the rate limit of initial sync for a new token can be temporarily increased within 24 hours only, but it may be not large enough for your case, I think. This may be the reason.

If you still have Evernote installed, and your Evernote has most of your notes synced, you can transfer locally.

If your notes consist of text mainly, and not too many multimedia files, you could consider to sync with Evernote and then transfer locally. I remember that Evernote client has a limit for data amount every month for free user, I am not sure if it is enough for your notes.

I don't have so many notes, and I don't know whether the option of 'Restart sync on API limit (experimental)' works or not in reality. But if nixnote doesn't restart the sync automatically, what about restarting nixnote every one hour with an outside shell commands or something like that, with the option of auto 'sync on start up' enabled? This method may take a long time, but it is available.

Thank you very much for your reply.
I' m on linux and evernote for linux has been disabled months ago. Until a couple of days ago I was using evernote legacy through wine and a repackage windows version that a guy was releasing on github here , and evernote disabled the API endpoint for both of them a couple of days ago.
That's how I landed here.
I still have both of them installed and I also have some recent backups in enex format.
I'm a subscribed user (not free user).
Should I import the backup notes on a fresh installation of nixnote and then pull down the new notes through a server sync?
Frankly I'm very afraid of doing a mess with the notes on the server, I keep very important information to me there.

The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that.
But, as I' m not using nixnote since many many years, would be all this syncing/importing work worth it? Is the client completely functional at this point?
It would be a great way to be able to keep using evernote on linux (other than migrating to a different platform).
Thank you again

[boo-yee]

Until a couple of days ago I was using evernote legacy through wine and a repackage windows version that a guy was releasing on github here

You were using evernote legacy through wine, so the latest evernote client can't work with wine, right?

I still have both of them installed and I also have some recent backups in enex format.

If there are only a few unsynced notes, how about using the web evernote and backing them up manually?

But, as I' m not using nixnote since many many years, would be all this syncing/importing work worth it?

You have backup files, so you don't need to sync all of them, backing up the unsynced notes is enough. Once you get all the necessary notes you want, you could not only import them to nixnote but also other apps you like.

Should I import the backup notes on a fresh installation of nixnote and then pull down the new notes through a server sync? Frankly I'm very afraid of doing a mess with the notes on the server, I keep very important information to me there.
The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that.

If you decide to use nixnote or other apps which are integrated with Evernote cloud, then this is a safe way, it just takes a long time.

Is the client completely functional at this point?

Nixnote is functional in most of the time. Some bugs may appear, but not often. The ui may be a little legacy. and don't offer some modern, fancy functions just as other newer note apps such as obsidian.

[wokawoka]

Thank you very much for your reply.

Until a couple of days ago I was using evernote legacy through wine and a repackage windows version that a guy was releasing on github here

You were using evernote legacy through wine, so the latest evernote client can't work with wine, right?

Yes, the new evernote version doesn't run under wine.

I still have both of them installed and I also have some recent backups in enex format.

If there are only a few unsynced notes, how about using the web evernote and backing them up manually?

The backup is from about two weeks ago, and in the meanwhile I might have probably edited many old notes other than adding new ones.
Even if I'm sure that it's a viable option, I'm afraid of messing with imports/exports of data that is not updated.

But, as I' m not using nixnote since many many years, would be all this syncing/importing work worth it?

You have backup files, so you don't need to sync all of them, backing up the unsynced notes is enough. Once you get all the necessary notes you want, you could not only import them to nixnote but also other apps you like.

So my last backup is not very fresh, but it looks like that tools like this are still working. So maybe I could pull down a fresh enex backup with that tool and then import the notes in nixnotes (but, by the way, why this tool is able to download and export the notes and nixnotes can't do the same?)

Should I import the backup notes on a fresh installation of nixnote and then pull down the new notes through a server sync? Frankly I'm very afraid of doing a mess with the notes on the server, I keep very important information to me there.
The idea of a script killing and restarting nixnote every 20 minutes is also a good idea and I could try that.

If you decide to use nixnote or other apps which are integrated with Evernote cloud, then this is a safe way, it just takes a long time.

Yes, I agree, I would much prefer to do a clean sync from the client itself. Scripting a recurrent shutdown/restart of nixnote like you were suggesting could be a good option.

Is the client completely functional at this point?

Nixnote is functional in most of the time. Some bugs may appear, but not often. The ui may be a little legacy. and don't offer some modern, fancy functions just as other newer note apps such as obsidian.

I'm not concerned about the look of the UI (and as I said, I was using the legacy version of evernote which was really looking oldshool, but it was quite snappy and functional). What I'm afraid of is inconsistencies between notes written in nixnote and in the new evernote official client. Recently I was using a mix a evernote legacy on one machine and evernote 10 on another machines and very often I was stepping into notes that were missing information in one client or in the other (it was happening especially with image attachments). I guess that overtime they changed things in their architecture and this is one of the reasons why they also dismissed the possibility to use the old clients.

[boo-yee]

So my last backup is not very fresh, but it looks like that tools like this are still working. So maybe I could pull down a fresh enex backup with that tool and then import the notes in nixnotes

If this tool works and has no rate limit, it will be easy to solve the problem.

(but, by the way, why this tool is able to download and export the notes and nixnotes can't do the same?)

I have not studied the code, I just read the README page, and I guess that it syncs the notes by calling the offical Evernote client APIs. The offical Evernote client may call another group of APIs, which are different from the third-party ones called by nixnote, and have no rate limit, I guess.

What I'm afraid of is inconsistencies between notes written in nixnote and in the new evernote official client. Recently I was using a mix a evernote legacy on one machine and evernote 10 on another machines and very often I was stepping into notes that were missing information in one client or in the other (it was happening especially with image attachments). I guess that overtime they changed things in their architecture and this is one of the reasons why they also dismissed the possibility to use the old clients.

Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.

[wokawoka]

If this tool works and has no rate limit, it will be easy to solve the problem.

I pulled a backup with that tool yesterday night and it correctly updated the backup I previously did last month (he even just downloaded the new and changed notes). Just great

I have not studied the code, I just read the README page, and I guess that it syncs the notes by calling the offical Evernote client APIs. The offical Evernote client may call another group of APIs, which are different from the third-party ones called by nixnote, and have no rate limit, I guess.

I guess that integrating their syncing method or the tricks they are using would benefit this project a lot.
At this stage the barrier of entry for someone who is using evernote since a long time and has to import a lot of notes (hitting the API limit) is too high

Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.

That makes me think that issues and inconsistencies might appear if someone is using nixnote and at the same time evernote 10 on other machines or on mobile clients

[d1vanov]

Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.

Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently.
Here's the specification for their ENEX format: http://xml.evernote.com/pub/evernote-export3.dtd.

What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such.

[wokawoka]

Evernote changed part of the note content format these years. If evernote 10 note format is not compatible with legacy client, then it's very possible that evernote 10 is not compatible with nixnote, because nixnote note format is designed to be compatible with legacy client, and did not follow the development and changes of evernote these years.

Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently. Here's the specification for their ENEX format: http://xml.evernote.com/pub/evernote-export3.dtd.

What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such.

I have no source but I have written what is my direct experience.

After many years of exclusively using the legacy version of evernote (under wine on linux), in the last 6 months I started using it alongside evernote 10 (on other machines), evernote android client, and evernote web.
Over the course of these months I ran into a lot of problems with synchronisation (especially on the evernote legacy side), problems with attachments (especially images which could no more be displayed natively in the client but could be downloaded) and even situations where whole chunks of text were missing.

This didn't happen when I was using legacy evernote exclusively.
My experience in these recent months has been that the two versions of evernote were less and less interoperable.
If the enex protocol will certainly have remained the same other things may have probably changed.

[boo-yee]

I pulled a backup with that tool yesterday night and it correctly updated the backup I previously did last month (he even just downloaded the new and changed notes). Just great

It is nice.

I guess that integrating their syncing method or the tricks they are using would benefit this project a lot.
At this stage the barrier of entry for someone who is using evernote since a long time and has to import a lot of notes (hitting the API limit) is too high

I see that the API rate limit was often talked about on github before, if there are not too many notes, just waiting and restarting the sync can solve it. In recent years, I feel that the users are not as many as before, so I think the improvement on syncing will not be needed by too many people.

That makes me think that issues and inconsistencies might appear if someone is using nixnote and at the same time evernote 10 on other machines or on mobile clients

I am afraid so.

[boo-yee]

@d1vanov

Any source? I haven't heard of them changing anything. Considering that they still have lots of business API integrations, it seems unlikely such moves would be done, especially silently.
Here's the specification for their ENEX format: http://xml.evernote.com/pub/evernote-export3.dtd.

What they could have actually changed is many other things which are present in note metadata but not in the note content (i.e. ENEX) itself - like all these reminder things and such.

What you said is reasonable, Evernote should be very serious about the format. But there are some posts about the note format on the evernote forum. For example, this link: https://discussion.evernote.com/forums/topic/148763-old-notes-bulleted-and-numbered-lists-messed-up/. To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons. To find the source Evernote released, I searched just now, and found this link: https://evernote.com/blog/legacy-decommissioning#:~:text=Legacy%20versions%20of%20Evernote%20use,and%20update%20at%20lightning%20speed.
This link says: Expect a slower experience the first time you open a note in v10. Legacy versions of Evernote use a different note format than v10. As such, each note must be converted to the new format. This action takes place when the note is opened for the first time in v10 and can last a couple of seconds. Going forward, it’ll load and update at lightning speed.

Maybe the word I used and my understanding are not precise enough, but it seems that there is incompatibility between the legacy client and the latest one.

[wokawoka]

I see that the API rate limit was often talked about on github before, if there are not too many notes, just waiting and restarting the sync can solve it. In recent years, I feel that the users are not as many as before, so I think the improvement on syncing will not be needed by too many people.

the situation with the API syncing is making it very very difficult to switch to nixnote for any user that has more than 2k notes and above stored on evernote servers.
That's a pity, and it is certainly a reason that keeps nixnote's user base low, even on a platform like linux where at the moment there is a desperate need for an evernote local client

[d1vanov]

To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons

Well, I don't know for certain either but I believe what has actually changed is not the ENEX format spec itself but the note editor implementation between legacy Evernote client and v10 and in how each of them interprets the format. ENEX in its core is just a stripped down XHTML which in turn is just a little more strict and XML-like HTML. So it all kinda boils down to how a web browser implements a rich text editor. V10 might use different HTML tags for lists than the legacy client used. But I don't think a change in ENEX spec was necessary for that to happen.

[wokawoka]

To this kind of issues, I just guessed it was because of the format changes, because except the note format changes, I cannot think out other reasons

Well, I don't know for certain either but I believe what has actually changed is not the ENEX format spec itself but the note editor implementation between legacy Evernote client and v10 and in how each of them interprets the format. ENEX in its core is just a stipped down XHTML which in turn is just a little more strict and XML-like HTML. So it all kinda boils down to how a web browser implements a rich text editor. V10 might use different HTML tags for lists than the legacy client used. But I don't think a change in ENEX spec was necessary for that to happen.

Yes, it makes sense